Hallucinations: why confident answers can be wrong

In 2023, a New York lawyer named Steven Schwartz submitted a legal brief citing six court cases: *Varghese v. China Southern Airlines*, *Martinez v. Delta Air Lines*, and four others. They had names, docket numbers, quotes, and judges. They looked perfect.

They were also completely fake. ChatGPT had invented every one of them. The judge fined Schwartz and his colleague $5,000, and the story went around the world.

Here is the unsettling part: ChatGPT did not "lie." It did exactly what it was built to do. Understanding why is the single most useful thing you can learn about AI.

What a hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → actually is

A hallucination is when an AI model produces text that is fluent, confident, and wrong. Not "wrong" like a typo. Wrong like inventing a court case, a research paper, a quote, or a statistic that never existed.

The word is a little misleading. The model is not "seeing things." It is generating the most plausible-sounding next words, and sometimes the most plausible-sounding answer is not the true one.

Why it happens

A large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → (an "LLM," the technology behind ChatGPT, Claude, and Gemini) does one core thing: it predicts the next word, over and over, based on patterns it learned from huge amounts of text.

It does not have a database of facts it looks things up in. It has *patterns*. When you ask for a legal citation, it knows what a citation looks like: a case name, "v.", two parties, a court, a year. So it produces something shaped exactly like a real citation, whether or not that case exists.

Think of it like a brilliant improv actor. Ask for a 1990s Bulgarian sci-fi film and they will confidently name one, describe the plot, and praise the lead actor. The performance is flawless. The film is fiction.

Why they sound so confident

This is what fooled the lawyer, and it fools everyone at first.

LLMs are trained to produce clear, fluent, authoritative-sounding text. They are not trained to express doubt accurately. A model has no built-in sense of "I actually know this" versus "I am guessing." Both come out in the same smooth, professional voice.

So a made-up citation reads exactly like a real one. There is no flicker, no hesitation, no "I think but I'm not sure." Fluency is not the same as accuracy, but our brains treat them as the same thing.

This is why "it sounded so sure" is the worst possible reason to trust an answer.

Where hallucinations are most dangerous

Some questions are far riskier than others. Watch out when you ask for:

• Specific citations, sources, or URLs. Models love to invent plausible-looking links and paper titles.
• Exact numbers, dates, and statistics. "73% of companies" sounds precise. It is often guessed.
• Quotes attributed to real people. Easy to fabricate, hard to spot.
• Legal, medical, or financial specifics. High stakes, and the model has no license.
• Recent events outside the model's training, unless it is actively searching the web.
• Niche or obscure topics where there was little training data to learn from.

The pattern: the more specific and verifiable a claim is, the more you should check it.

A real example you can try

Ask any chatbot this:

"Give me three peer-reviewed studies on the effect of houseplants on office productivity, with authors and journal names."

You will often get clean, confident, beautifully formatted citations. Then try to find them on Google Scholar. Some may be real. Some may have real authors but a fake title. Some will not exist at all. This single exercise teaches the lesson better than any warning.

The simple habits that catch it

You do not need to be technical to protect yourself. You need a few reflexes.

1. Treat AI output as a draft, not a verdict

The model gives you a starting point. You are the editor. For anything that matters, verify before you use it.

2. Ask for sources, then actually check them

Asking "what's your source?" helps a little, but the model can invent sources too. The real step is clicking through. If a link does not open or a case does not appear in a real database, drop the claim.

3. Use tools that search the web

In 2026, ChatGPT, Claude, and Gemini can all browse the web and cite live pages. This dramatically reduces hallucination on current facts because the model is reading real sources instead of recalling patterns. Turn this on for anything factual:

• In ChatGPT, use search mode or the research tools.
• In Claude, enable web search.
• In Gemini, it pulls from Google Search and shows links.

Always click the links it shows. A cited source is only useful if it actually says what the model claims.

4. Ask the model to flag its own uncertainty

This prompt helps:

"Answer the question. Then list which specific claims you are confident about and which you are unsure about or may be guessing."

It is not perfect, but it often surfaces the shaky parts.

5. Cross-check across two models

If ChatGPT and Gemini independently give the same answer, your confidence can go up. If they disagree, you have found exactly the spot to investigate.

A light technical habit: ground the answer

If you ever build with the AI tools (through their developer interface, called an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →, a way for programs to send requests to the model), you can cut hallucinations by giving the model the source text yourself. This is called grounding: instead of asking the model to recall, you hand it the facts and tell it to answer only from those.

Here is the idea in a short Python snippet using the OpenAI API:

from openai import OpenAI

client = OpenAI()

source_text = """
Our return policy: customers can return items within 30 days
for a full refund. Electronics must be returned within 14 days.
"""

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "system",
            "content": (
                "Answer ONLY using the provided policy text. "
                "If the answer is not in the text, say 'Not specified.'"
            ),
        },
        {"role": "user", "content": source_text + "\nHow long do I have to return a laptop?"},
    ],
)

print(response.choices[0].message.content)

The instruction "if the answer is not in the text, say 'Not specified'" gives the model permission to admit it does not know. That single line prevents a lot of confident guessing. The model should answer "14 days" here, and refuse to invent a policy you never gave it.

You do not need to code to use this idea. Even in the chat window, pasting in the actual document and saying "answer only from this text" works the same way.

What hallucinations are not

A quick clarification, because people often blame the wrong thing.

A hallucination is not a bug that will get patched away next month. It is a side effect of how these models work. Newer models in 2026 hallucinate less, especially when searching the web, but no model is at zero. Build your habits as if it can always happen, because it can.

It is also not the model being malicious. There is no intent. It is pattern-completion that landed on a confident falsehood.

Bringing it back to the lawyer

Schwartz's real mistake was not using ChatGPT. Plenty of lawyers use AI well. His mistake was trusting confident output on a high-stakes, highly verifiable task without checking a single citation.

One search on a legal database would have caught it in minutes. The tool was fine. The missing habit was verification.

That is the whole lesson: the AI is a fast, fluent, sometimes-wrong assistant. Your judgment is the safety check.

Key Takeaways

• Fluency is not accuracy. A confident, polished answer can be completely fabricated. Never trust tone as evidence.
• Verify anything specific and high-stakes: citations, numbers, quotes, links, and legal, medical, or financial claims. Click through to a real source.
• Turn on web search in ChatGPT, Claude, or Gemini for factual questions, and check the links it cites instead of trusting them blindly.
• Ground the model by pasting in the actual source text and instructing it to answer only from that, and to say "not specified" when the answer is missing.
• Treat every AI answer as a draft you edit, not a verdict you publish. You are the final check, every time.