Hallucinations and verification in high-stakes work

In 2023, two New York lawyers were fined $5,000 after they filed a legal brief citing six court cases that did not exist. ChatGPT had invented them, complete with fake quotes and fake case numbers. The lawyers did not check. The judge did.

That is the core problem in one story. AI models can produce confident, fluent, completely false information. We call this a hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition →: when a model generates content that sounds right but is factually wrong or entirely made up.

A made-up statistic in a brainstorm costs you nothing. The same error in a medical chart, a contract, or a tax filing can end a career or harm a person. This lesson teaches you to match your verification effort to the stakes.

Why hallucinations happen

A large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → (the AI behind ChatGPT, Claude, and Gemini) is a prediction engine. It predicts the next likely word based on patterns in its training data. It is not looking up facts in a database. **It is producing text that *resembles* a correct answer**.

This is why hallucinations are so dangerous: the model has no internal alarm that says "I'm guessing now." A real citation and a fabricated one look equally confident.

Models in 2026 are better than they were. Tools like ChatGPT, Claude, and Gemini now cite sources and search the web. But "better" is not "solved." The fabrication rate dropped; it did not hit zero.

The tiered verification mindset

You do not need to fact-check every word an AI gives you. That would make AI useless. Instead, ask one question before you act on any output:

What happens if this is wrong?

The answer sets your verification tier.

Tier 1: low stakes (glance and go)

The cost of an error is trivial or easily reversed.

• Brainstorming blog titles
• Drafting a casual email you will reread anyway
• Summarizing an article for your own understanding
• Generating ideas for a birthday party

Verification: A quick read for obvious nonsense. That is it. If a brainstormed statistic feels off, drop it or check it later.

Tier 2: medium stakes (spot-check the claims)

An error would be embarrassing, cost real time, or mislead colleagues, but is not dangerous or legally binding.

• A market summary for an internal team meeting
• A how-to guide you will publish on a company blog
• Product comparison research for a purchase decision
• A grant application draft

Verification: Check every specific, checkable claim. Names, numbers, dates, quotes, and statistics. Open a second source for anything you will state as fact.

A simple habit: ask the model to separate facts from interpretation.

"List every factual claim in your last answer as a bullet list, with the specific source for each. Mark anything you are not confident about."

Then verify the bullets, not the prose.

Tier 3: high stakes (verify everything, independently)

An error could harm a person, break the law, lose significant money, or destroy trust.

• Medical: drug dosages, diagnoses, drug interactions
• Legal: case citations, contract clauses, statutory deadlines
• Financial: tax advice, investment figures, regulatory filings
• Safety: instructions involving electricity, chemicals, structural work

Verification: Treat the AI output as a *draft by an unverified intern*. Every single fact must be confirmed against an authoritative, independent source. A doctor checks a drug database. A lawyer pulls the actual case. An accountant confirms against current tax code.

In Tier 3, the AI does not get the final word. A qualified human and a primary source do.

Concrete Scenarios

Medical. You ask an AI: "What is the max daily dose of acetaminophen for an adult?" It answers "4,000 mg." That happens to be a common figure, but doses depend on liver health, alcohol use, and other medications. In a clinical setting, you confirm against a source like the FDA label database or a clinical reference, never the chatbot alone.

Legal. You ask for cases supporting your argument. The model returns three with perfect-looking citations. Before anything goes in a filing, you look up each case on the actual court database or a service like CourtListener. If you cannot find it, it does not exist.

Financial. An AI tells you a tax deduction limit. Tax rules change yearly and vary by region. You confirm against the current official guidance (for the US, IRS.gov) or a licensed professional.

The pattern is identical: the AI accelerates the draft, but the source of truth is never the AI.

Reducing hallucinations before you verify

Verification is your safety net. Good prompting shrinks the number of errors you have to catch.

1. Give the model the source material. If you paste the actual document and ask questions about it, the model has far less room to invent. This is called grounding.

2. Tell it to admit uncertainty. Add: "If you are not sure or do not have a reliable source, say so. Do not guess." Modern models respect this instruction well.

3. Use tools that cite. In 2026, Gemini, ChatGPT, and Claude all offer web search modes with clickable sources. Turn them on for factual work. Then *click the sources*. A citation you do not open is not verification.

4. Ask for the reasoning, then check it. A wrong conclusion often has a visibly wrong step.

Here is a tiny example using an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → to force a "confidence and sources" structure into the output:

from openai import OpenAI

client = OpenAI()

prompt = """Answer the question below.
Then add two sections:
FACTS: every checkable claim as a bullet, each with a source.
CONFIDENCE: low / medium / high, and say what would change your answer.

Question: What is the standard statute of limitations for
breach of written contract in California?"""

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
)

print(response.choices[0].message.content)

The structured output makes verification fast: you check the FACTS bullets and treat anything marked "low" confidence as unverified until proven.

Building a verification habit

The biggest risk is not the model. It is automation bias: our human tendency to trust a confident, well-formatted answer just because a machine produced it. Polished formatting feels like accuracy. It is not.

Build small rituals that interrupt that reflex.

The 3-question gate

Before any AI output leaves your hands, ask:

1. What tier is this? (What happens if it's wrong?)
2. Which specific claims are checkable? (Names, numbers, dates, citations, quotes.)
3. Have I confirmed those against an independent source?

For Tier 1, this takes ten seconds. For Tier 3, it is the whole job.

Watch for the hallucination red flags

Certain outputs deserve extra suspicion:

• Specific citations and URLs. Models fabricate these often. Always click.
• Precise statistics with no source. "73% of users prefer..." Where is that from?
• Recent events. Models can be out of date or confabulate current details.
• Niche or obscure topics. Less training data means more guessing.
• **Anything you *want* to be true.** Your own bias plus the model's fluency is a dangerous mix.

Make someone accountable

In a team or organization, name a human owner for every high-stakes AI output. "The AI wrote it" is not a defense. The New York lawyers learned that the hard way. Someone signs off, and that someone checked.

For a deeper, practical framework on this, the NIST AI Risk Management Framework is a free, well-respected resource on managing AI risk in real organizations.

Putting it together: a quick workflow

Say you are writing a customer-facing FAQ about a financial product.

1. Draft with AI. Paste your real product docs and ask it to write the FAQ from those docs only.
2. Tier it. Customer-facing financial info is Tier 3.
3. Extract claims. Ask the model to list every factual statement with its source from your docs.
4. Verify independently. Confirm each against the source documents and current regulations.
5. Human sign-off. A qualified person reviews and approves before publishing.

The AI did 70% of the work fast. You did the 30% that protects everyone.

Key Takeaways

• Match verification to stakes. Ask "What happens if this is wrong?" before you act. Glance-and-go for low stakes; verify everything independently for medical, legal, and financial work.
• The AI is never the source of truth in Tier 3. Confirm every fact against an authoritative, independent source, and have a qualified human sign off.
• Watch the red flags. Citations, URLs, precise statistics, recent events, and niche topics are where hallucinations cluster. Click every source; do not trust formatting.
• Reduce errors before you catch them. Ground the model in real source material, tell it to admit uncertainty, and use search modes that cite, then actually open the citations.
• Beware automation bias. A confident, polished answer is not a verified one. Name a human owner for high-stakes outputs.