RAG in production — how retrieval-augmented generation works, and where it breaks

In one paragraph

RAG — retrieval-augmented generation — means giving a model the right context at answer time instead of hoping it memorized it. You retrieve the relevant documents, put them in the prompt, and let the model answer grounded in them, ideally citing what it used. It’s how you get an LLM to answer over your private and current data without retraining it. The catch is simple and it’s where most projects go wrong: a RAG system is only as good as what it retrieves. When the answer is wrong, the model usually isn’t the problem. Retrieval is.

Why RAG exists

A language model is frozen at training time, and it’ll make things up confidently when it doesn’t know. That creates two gaps. It doesn’t know your private data — your contracts, your tickets, your policies. And it doesn’t know what changed after its training cut-off. Retraining or fine-tuning to close those gaps is slow, expensive, and goes stale the moment a document changes.

RAG sidesteps both. Keep the model fixed, and at query time fetch the facts that matter and hand them to the model as context. The original RAG paper (Lewis et al., 2020) framed this for knowledge-intensive tasks; six years on, it’s the default way to put an LLM on top of a body of documents.

How it actually works

The pipeline has a few stages, and each one is a place to get it right or wrong.

• Ingest and chunk. Documents are split into passages small enough to retrieve precisely but large enough to keep their meaning.
• Embed. Each chunk is turned into a vector and stored in an index, so “similar in meaning” becomes “near in vector space.”
• Retrieve. The query is embedded and the nearest chunks are pulled — in practice alongside a keyword search, because dense vectors and exact-term matching each catch what the other misses.
• Rerank. A second, heavier model reorders the top candidates by actual relevance to the query, not just vector proximity.
• Generate. The top chunks and the query go to the LLM, which answers grounded in them and cites its sources.

This is the same shape that sits behind public AI answer engines like Perplexity: retrieve, rerank, then synthesize with inline citations. The citation is chosen at retrieval time, before any sentence is written — a teardown of Perplexity’s pipeline describes citation markers being assigned during context assembly, not retrofitted after the model writes.

Where it is used now

RAG has quietly become the standard way to ship LLM features over real data:

• Enterprise search and Q&A over internal wikis, policies, contracts, and tickets — ask a question, get a grounded answer with links to the source.
• Customer support, answering from your own knowledge base instead of a generic model’s guesses.
• Document processing and extraction, pulling fields from statements and forms, where retrieval finds the right clause and the model reads it.
• Coding assistants, retrieving the relevant files and docs before they suggest a change.
• Agentic workflows, where retrieval is one tool among several an agent calls — deciding when and what to fetch before acting, instead of running a single fixed query.

Where it breaks

The hard parts of RAG are almost never the model. They are in the retrieval and the plumbing around it.

• Retrieval is the bottleneck. If the right passage isn’t in what you retrieved, the model can’t use it, and a confident wrong answer is worse than no answer. Practitioners who have run RAG in production trace most failures back to retrieval, not generation — when the evidence is incomplete or poorly ranked, even a strong model produces a weak answer. Garbage in, garbage cited.
• Chunking is a real decision. Chunks too large bury the answer in noise; too small and a passage loses the context that made it meaningful. There’s no universal setting — it depends on your documents.
• The retriever has biases. Neural retrievers over-favor low-perplexity text — fluent, predictable writing. The Perplexity Trap study showed they learn to treat low perplexity as a stand-in for relevance, so a smooth-but-wrong passage can outrank a correct-but-clunky one.
• A stale index lies confidently. If you don’t re-index when documents change, RAG will cite last quarter’s policy with full conviction.
• It has an attack surface. If your index ingests untrusted content, corpus poisoning is a real risk — PoisonedRAG (USENIX Security 2025) showed that injecting five crafted texts per target question into a corpus of millions can hit a 90% attack success rate, steering the answer to whatever the attacker chose.
• You can’t eyeball quality. RAG needs a labelled evaluation set with retrieval metrics and answer metrics, gated in CI, or you won’t notice when a change quietly makes it worse.

How we build it

The way we ship RAG into a custom system is the same discipline as the rest of our automation work: measure, gate, cite, and audit.

• Hybrid retrieval plus a reranker. Dense vectors and keyword search together, fused and then reordered by a cross-encoder reranker, because recall and precision both decide whether the right passage reaches the model.
• Evals gated in CI. Quality is a measured number against a labelled set, not a vibe; retrieval and answer quality are scored separately so you can see which half regressed, with frameworks like RAGAS for the answer metrics and Playwright for the end-to-end checks where a pipeline drives a real interface.
• Confidence gating and citations. The system answers only when retrieval supports it, shows its sources, and escalates or asks rather than guessing when it does not.
• Fresh index, untrusted input. We re-index when content changes, and treat ingested content as untrusted, with field access enforced server-side rather than filtered in the prompt.

Why this matters now

RAG is how most enterprises will actually put LLMs to work on their own data — answers grounded in current sources and auditable after the fact, without betting the business on a model’s memory. Gartner expects AI agents to outnumber human sellers tenfold by 2028, with agentic systems increasingly mediating B2B buying — and those agents evaluate structured, verifiable data, not a sales pitch. The teams that get value from that shift will be the ones whose retrieval is good and whose pipelines can prove what they answered and why.

References

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the original RAG paper that framed retrieve-then-generate (Lewis et al., 2020).
2. How Perplexity AI Answers Work: Retrieval, Ranking, and Citation Pipeline — teardown showing citations are assigned at retrieval, before generation (ZipTie, 2025).
3. Reasoning RAG via System 1 or System 2: A Survey on Agentic RAG — survey of agentic RAG, where retrieval is one tool the agent decides to call (Singh et al., 2025).
4. Why RAG Systems Fail in Production — argues most RAG failures originate in retrieval, not generation (DigitalOcean, 2025).
5. Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents — causal study of retrievers favoring fluent, low-perplexity text (ICLR 2025).
6. PoisonedRAG: Knowledge Corruption Attacks to RAG — five injected texts per question reach a 90% attack success rate (Zou et al., USENIX Security 2025).
7. Hybrid Search: BM25, Vector and Reranking — reference for combining sparse + dense retrieval with RRF and cross-encoder reranking (Digital Applied, 2026).
8. Ragas: Automated Evaluation of Retrieval Augmented Generation — separates retrieval and generation metrics for RAG evaluation (Es et al., 2023).
9. Playwright — end-to-end browser automation used to drive pipeline checks against a real interface.
10. Gartner: AI agents will outnumber sellers 10x by 2028 — Gartner forecast on agentic AI in B2B buying (Gartner, Nov 2025).