Why most RAG deployments fail before they go live

A legal team at a mid-sized financial services firm spends four months building a RAG system to help analysts query internal compliance documents. The demo works beautifully. Six weeks after go-live, usage has dropped by 70%. Analysts say the answers "feel wrong" and they've gone back to keyword search. The model was fine. The retrieval pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → was the problem, and nobody had measured it separately.

This story, with minor variations in industry and document type, plays out constantly across enterprise AI programs in 2026. RAG has become the dominant architecture for bringing LLMs into organizations with sensitive or specialized data, but the failure rate in production remains stubbornly high. Understanding why requires separating the three distinct places where these systems actually break.

Where RAG actually breaks

Most teams treat RAG as a single system to evaluate holistically. That framing is the first mistake. RAG is a pipeline with at least three independent failure modes: the retrieval layer, the context assembly, and the generation step. Conflating them makes diagnosis nearly impossible.

Retrieval is the silent killer

The LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → at the end of a RAG pipeline is usually the least problematic component. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are genuinely capable of synthesizing well-formed answers when given good context. The retrieval layer, by contrast, is where most production systems quietly fall apart.

The core issue is that dense vectordense vectorAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition → search, which became the default retrieval method as embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition → models improved, works well for semantic similarity but fails on precision tasks. Ask "what was our Q3 2024 gross margingross marginGross margin is the share of revenue left after subtracting the direct cost of producing goods or services, expressed as a percentage of revenue.View full definition →?" and a cosine similarity search across a vector storevector storeA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.View full definition → of financial documents will often return passages that are semantically adjacent but numerically incorrect, or from the wrong time period entirely. The model then synthesizes a confident, fluent, wrong answer.

Hybrid retrieval, combining dense vector search with BM25 or other sparse keyword methods, closes much of this gap. Cohere, Weaviate, and Elastic have all published benchmarks showing meaningful recall improvements on domain-specific corpora with hybrid approaches, though these are vendor figures and should be validated against your own document distribution before treating them as guarantees.

The chunking strategy compounds retrieval failures in ways that are rarely visible until production. Fixed-size chunking at 512 or 1024 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →, the default in most tutorials and starter kits, routinely cuts sentences mid-thought and severs tables from their headers. A retrieved chunk that says "increased by 14% compared to the prior period" is nearly useless without the surrounding context that establishes what increased and which prior period. Semantic chunking, which attempts to split on meaning boundaries rather than tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → counts, reduces this problem significantly but adds processing complexity and cost.

Context assembly: the problem nobody talks about

Assume retrieval works correctly and returns the five most relevant passages. The next failure point is how those passages are assembled into the prompt. Order matters more than most practitioners expect. Research from teams at Stanford and DeepMind has documented the "lost in the middle" phenomenon: LLMs systematically underweight information positioned in the middle of long contexts compared to content near the beginning or end. If your most relevant retrieved chunk lands in position three of five, the model may effectively discount it.

Re-ranking retrieved passages before assembling the prompt, using a cross-encoder model like those from Cohere (a vendor with a direct commercial interest in this recommendation) or open-source alternatives like BGE-Reranker, addresses this by reordering chunks by relevance before the LLM sees them. The latency cost is real, typically 100 to 300ms additional per query, and that tradeoff needs to be made deliberately.

Generation failures are usually retrieval failures in disguise

When users report that a RAG system "makes things up," the root cause is almost always retrieval returning irrelevant or insufficient context. The model is not hallucinating in the colloquial sense; it is doing exactly what it was designed to do, which is generate a plausible completion given the available context. If that context is poor, the completion will be confidently wrong.

The fix is not prompt engineeringprompt engineeringPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.View full definition →. Adding "only use the provided documents" to a system prompt does reduce hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → rates on clean retrievals, but it does almost nothing when the retrieved context is genuinely misleading or incomplete. Measuring retrieval quality independently, using metrics like context precision and context recall from the RAGAS evaluation framework, is the only way to know which layer is actually failing.

What this means for teams building or buying RAG

If your organization is deploying a RAG system, the evaluation architecture matters as much as the system architecture itself. Before launch, establish separate benchmarks for retrieval quality and generation quality. Run them against a representative sample of real user queries, not synthetic ones designed by the team that built the system.

Document preprocessing deserves more investment than it typically receives. PDFs with complex layouts, scanned documents, and tables embedded in slides are where most enterprise knowledge actually lives, and they are where standard ingestion pipelines perform worst. Tools like LlamaParse (from LlamaIndex, a vendor) and AWS Textract handle structured extraction more reliably than generic PDF parsers, but neither is a complete solution for mixed-format document libraries.

For teams evaluating vendor-built RAG products: ask specifically how the product handles retrieval failures, not just hallucinations. Any vendor can demonstrate a clean demo. Ask to see behavior when the retrieved context is ambiguous or contradictory, because that is what your production environment will look like most of the time.

Practical actions before your next sprint

• Instrument your retrieval layer independently from generation. Log what was retrieved for every query, not just the final answer.
• Run at least 50 real user queries through the system before launch and manually inspect the retrieved chunks, not the generated outputs.
• If your documents contain tables, dates, or numeric data, test hybrid retrieval against pure vector search on those specific query types before defaulting to either.
• Set a latency budget early and include re-ranking in that budget calculation, since it is the step most commonly dropped under time pressure and the one that most affects answer quality.
• Treat "the model hallucinated" as a starting hypothesis, not a conclusion. Trace the failure back through the retrieval step before making any changes to prompts or model parameters.

The compliance team in that financial services firm eventually fixed their system, not by switching models, but by adding BM25 hybrid retrieval and moving to semantic chunking. Usage recovered within three weeks of the redeployment. The fix was technical, but the insight that unlocked it was organizational: someone finally measured retrieval and generation as separate problems.