# Hallucinations and verification in high-stakes work
In 2023, two New York lawyers were fined $5,000 after they filed a legal brief citing six court cases that did not exist. ChatGPT had invented them, complete with fake quotes and fake case numbers. The lawyers did not check. The judge did.
That is the core problem in one story. AI models can produce confident, fluent, completely false information. We call this a hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.Voir la définition complète →: when a model generates content that sounds right but is factually wrong or entirely made up.
A made-up statistic in a brainstorm costs you nothing. The same error in a medical chart, a contract, or a tax filing can end a career or harm a person. This lesson teaches you to match your verification effort to the stakes.
A large language model (the AI behind ChatGPT, Claude, and Gemini) is a prediction engine. It predicts the next likely word based on patterns in its training data. It is not looking up facts in a database. It is producing text that *resembles* a correct answer.
This is why hallucinations are so dangerous: the model has no internal alarm that says "I'm guessing now." A real citation and a fabricated one look equally confident.
Models in 2026 are better than they were. Tools like ChatGPT, Claude, and Gemini now cite sources and search the web. But "better" is not "solved." The fabricationfabricationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.Voir la définition complète → rate dropped; it did not hit zero.
You do not need to fact-check every word an AI gives you. That would make AI useless. Instead, ask one question before you act on any output:
What happens if this is wrong?
The answer sets your verification tier.
The cost of an error is trivial or easily reversed.
Verification: A quick read for obvious nonsense. That is it. If a brainstormed statistic feels off, drop it or check it later.
An error would be embarrassing, cost real time, or mislead colleagues, but is not dangerous or legally binding.
Verification: Check every specific, checkable claim. Names, numbers, dates, quotes, and statistics. Open a second source for anything you will state as fact.
A simple habit: ask the model to separate facts from interpretation.
> "List every factual claim in your last answer as a bullet list, with the specific source for each. Mark anything you are not confident about."
Then verify the bullets, not the prose.
An error could harm a person, break the law, lose significant money, or destroy trust.
Verification: Treat the AI output as a *draft by an unverified intern*. Every single fact must be confirmed against an authoritative, independent source. A doctor checks a drug database. A lawyer pulls the actual case. An accountant confirms against current tax code.
In Tier 3, the AI does not get the final word. A qualified human and a primary source do.
Medical. You ask an AI: "What is the max daily dose of acetaminophen for an adult?" It answers "4,000 mg." That happens to be a common figure, but doses depend on liver health, alcohol use, and other medications. In a clinical setting, you confirm against a source like the FDA label database or a clinical reference, never the chatbot alone.
Legal. You ask for cases supporting your argument. The model returns three with perfect-looking citations. Before anything goes in a filing, you look up each case on the actual court database or a service like CourtListener. If you cannot find it, it does not exist.
Financial. An AI tells you a tax deduction limit. Tax rules change yearly and vary by region. You confirm against the current official guidance (for the US, IRS.gov) or a licensed professional.
The pattern is identical: the AI accelerates the draft, but the source of truth is never the AI.
Verification is your safety net. Good promptingpromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.Voir la définition complète → shrinks the number of errors you have to catch.
1. Give the model the source material. If you paste the actual document and ask questions about it, the model has far less room to invent. This is called grounding.
2. Tell it to admit uncertainty. Add: "If you are not sure or do not have a reliable source, say so. Do not guess." Modern models respect this instruction well.
3. Use tools that cite. In 2026, Gemini, ChatGPT, and Claude all offer web search modes with clickable sources. Turn them on for factual work. Then *click the sources*. A citation you do not open is not verification.
4. Ask for the reasoning, then check it. A wrong conclusion often has a visibly wrong step.
Here is a tiny example using an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.Voir la définition complète → to force a "confidence and sources" structure into the output:
from openai import OpenAI
client = OpenAI()
prompt = """Answer the question below.
Then add two sections:
FACTS: every checkable claim as a bullet, each with a source.
CONFIDENCE: low / medium / high, and say what would change your answer.
Question: What is the standard statute of limitations for
breach of written contract in California?"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
)
print(response.choices[0].message.content)The structured output makes verification fast: you check the FACTS bullets and treat anything marked "low" confidence as unverified until proven.
Vérification des acquis
1. In the 2023 case described in the lesson, what mistake did the two New York lawyers make with ChatGPT?
2. According to the lesson, why does a language model produce hallucinations with such convincing confidence?
3. Modern tools like ChatGPT, Claude, and Gemini now cite sources and search the web. What does the lesson say this capability achieves regarding hallucinations?
4. Select ALL correct answers. Which tasks fit the lesson's 'Tier 1: Low Stakes (Glance and Go)' category?
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers. Which statements accurately reflect the lesson's 'tiered verification mindset'?
Sélectionnez toutes les réponses correctes.
The biggest risk is not the model. It is automation bias: our human tendency to trust a confident, well-formatted answer just because a machine produced it. Polished formatting feels like accuracy. It is not.
Build small rituals that interrupt that reflex.
Before any AI output leaves your hands, ask:
1. What tier is this? (What happens if it's wrong?)
2. Which specific claims are checkable? (Names, numbers, dates, citations, quotes.)
3. Have I confirmed those against an independent source?
For Tier 1, this takes ten seconds. For Tier 3, it is the whole job.
Certain outputs deserve extra suspicion:
In a team or organization, name a human owner for every high-stakes AI output. "The AI wrote it" is not a defense. The New York lawyers learned that the hard way. Someone signs off, and that someone checked.
For a deeper, practical framework on this, the NIST AI Risk Management Framework is a free, well-respected resource on managing AI risk in real organizations.
Say you are writing a customer-facing FAQ about a financial product.
1. Draft with AI. Paste your real product docs and ask it to write the FAQ from those docs only.
2. Tier it. Customer-facing financial info is Tier 3.
3. Extract claims. Ask the model to list every factual statement with its source from your docs.
4. Verify independently. Confirm each against the source documents and current regulations.
5. Human sign-off. A qualified person reviews and approves before publishing.
The AI did 70% of the work fast. You did the 30% that protects everyone.