Why LLMs still confabulate, and what you should actually do about it

A lawyer submits a brief citing six precedents. Opposing counsel checks them. None exist. The cases have plausible names, realistic docket numbers, and coherent summaries, all generated by ChatGPT, none of them real. This incident from 2023 became a landmark cautionary tale, but three years later, the underlying dynamic that produced it has not been engineered away. Models have improved, retrieval mechanisms have been bolted on, and context windows have expanded, but the tendency to generate fluent, authoritative-sounding text that is factually wrong remains a structural feature of how these systems work.

This matters not just for lawyers but for anyone using LLMs to draft reports, synthesize research, or support decisions where the underlying facts are load-bearing.

Why confabulationconfabulationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → is not a calibration problem you can simply tune away

Language models are trained to predict the next tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → given a sequence of prior tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →. That objective, applied at massive scale, produces extraordinarily capable systems, but it also means the model's primary job is coherence and plausibility, not truth. When a model doesn't "know" something in any meaningful sense, it doesn't return a blank. It generates what would plausibly come next, which often looks like a confident, specific answer.

Retrieval-augmented generation (RAG) helps significantly. By grounding the model in retrieved documents at inference time, you reduce the surface area for confabulation on factual questions that are well-covered in the retrieval corpus. OpenAI's GPT-4o with web search, Anthropic's Claude with document attachments, Google's Gemini connected to Workspace data: all of these architectures push in the right direction. But RAG doesn't eliminate the problem. It relocates it. The model can still misread retrieved documents, blend content from multiple sources incorrectly, or confabulate when no retrieved passage quite covers the question being asked.

The deeper issue is that confabulation rates are not uniform across domains. Models perform significantly better on topics that were heavily represented in training data, where patterns are dense and consistent. Ask a model about the structure of a standard commercial lease or the mechanics of a discounted cash flowdiscounted cash flowDiscounted Cash Flow (DCF) is a valuation method that estimates an asset's value by projecting future cash flows and discounting them to present value using a required rate of return.View full definition → model and it will likely be accurate. Ask it about a niche regulatory change from 18 months ago, an obscure country-level tax rule, or the internal metrics of a specific private company, and you are in much higher-risk territory. Most professionals using LLMs in real work operate routinely in that second category.

What this means for the AI user

The practical implication is not "use LLMs less." It's "use them with explicit verification logic built into the workflow from the start, not added as an afterthought."

The first thing to internalize is that tone is not a reliability signal. Models are equally fluent when they are right and when they are wrong. The confident, well-structured paragraph is not evidence of accuracy. This is the failure mode that catches experienced professionals who have correctly learned to use formatting and tone as proxies for quality in human writing. That heuristic breaks completely with LLMs.

Second, verification effort should scale with the stakes and the obscurity of the domain. For widely documented topics where errors would be immediately visible to any reader, the risk is lower. For domain-specific, recent, or numerically precise claims, every key fact needs an independent source. Build that step into the workflow explicitly. If you are using an LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → to summarize analyst reports, check that cited statistics actually appear in the source document. If you are drafting a market overview, confirm that company figures come from filings or verified third-party databases, not from the model's recall.

Third, the architecture you choose changes the risk profile. A raw prompt to a general-purpose model with no retrieval carries the highest confabulation risk for factual tasks. A well-designed RAG pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → with high-quality, curated source documents and explicit citation requirements in the prompt reduces it substantially. If your use case is genuinely high-stakes and domain-specific, the infrastructure investment in a proper retrieval layer is not optional.

PromptingPromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.View full definition → for uncertainty, not just answers

One underused technique: explicitly instruct the model to flag its uncertainty. Prompts like "If you are not confident in any of the following claims, say so explicitly" or "List any assertions in your response that rely on information you cannot verify from the provided documents" produce more calibrated outputs. This doesn't eliminate confabulation, but it surfaces the model's own hesitation and creates a shorter list of things to verify.

Some teams have formalized this into a two-pass workflow. The first pass generates the draft. The second pass, using a separate prompt, asks the model to identify every specific factual claim in that draft and rate its confidence in each. The second pass regularly catches things the first pass presented with unwarranted confidence.

What to actually do

• Separate your LLM use cases by verification cost. Tasks where errors are recoverable or low-stakes (drafting tone, restructuring prose, brainstorming) require less overhead. Tasks where factual errors could propagate into decisions, contracts, or client deliverables require a verification step built into the process, not delegated to goodwill.
• Do not rely on the model's self-reported confidence when the output matters. Ask for citations, then check them. If the model cannot produce a verifiable source, treat the claim as unverified.
• For recurring high-stakes workflows, invest in retrieval architecture over prompting tricks. A RAG system built on your actual internal documents and verified external sources will outperform a clever prompt against a general model every time on factual accuracy within that domain.
• Train anyone on your team who uses LLMs to recognize that fluency and accuracy are orthogonal. This is a calibration shift that takes deliberate effort to internalize.
• When a model's answer to a niche question arrives immediately and with no hedging, treat that as a flag, not a green light. Legitimate domain expertise comes with caveats.

The legal brief story keeps getting cited because it captures the failure mode cleanly. A professional trusted a fluent output and skipped verification. The model did exactly what it was designed to do. Building reliable LLM workflows means designing around that reality, not hoping the next model release will make it irrelevant.