# Hallucinations: why confident answers can be wrong
In 2023, a New York lawyer named Steven Schwartz submitted a legal brief citing six court cases: *Varghese v. China Southern Airlines*, *Martinez v. Delta Air Lines*, and four others. They had names, docket numbers, quotes, and judges. They looked perfect.
They were also completely fake. ChatGPT had invented every one of them. The judge fined Schwartz and his colleague $5,000, and the story went around the world.
Here is the unsettling part: ChatGPT did not "lie." It did exactly what it was built to do. Understanding why is the single most useful thing you can learn about AI.
The word is a little misleading. The model is not "seeing things." It is generating the most plausible-sounding next words, and sometimes the most plausible-sounding answer is not the true one.
A large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → (an "LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition →," the technology behind ChatGPT, Claude, and Gemini) does one core thing: it predicts the next word, over and over, based on patterns it learned from huge amounts of text.
It does not have a database of facts it looks things up in. It has *patterns*. When you ask for a legal citation, it knows what a citation looks like: a case name, "v.", two parties, a court, a year. So it produces something shaped exactly like a real citation, whether or not that case exists.
Think of it like a brilliant improv actor. Ask for a 1990s Bulgarian sci-fi film and they will confidently name one, describe the plot, and praise the lead actor. The performance is flawless. The film is fiction.
This is what fooled the lawyer, and it fools everyone at first.
LLMs are trained to produce clear, fluent, authoritative-sounding text. They are not trained to express doubt accurately. A model has no built-in sense of "I actually know this" versus "I am guessing." Both come out in the same smooth, professional voice.
So a made-up citation reads exactly like a real one. There is no flicker, no hesitation, no "I think but I'm not sure." Fluency is not the same as accuracy, but our brains treat them as the same thing.
This is why "it sounded so sure" is the worst possible reason to trust an answer.
Some questions are far riskier than others. Watch out when you ask for:
The pattern: the more specific and verifiable a claim is, the more you should check it.
Ask any chatbot this:
> "Give me three peer-reviewed studies on the effect of houseplants on office productivity, with authors and journal names."
You will often get clean, confident, beautifully formatted citations. Then try to find them on Google Scholar. Some may be real. Some may have real authors but a fake title. Some will not exist at all. This single exercise teaches the lesson better than any warning.
You do not need to be technical to protect yourself. You need a few reflexes.
The model gives you a starting point. You are the editor. For anything that matters, verify before you use it.
Asking "what's your source?" helps a little, but the model can invent sources too. The real step is clicking through. If a link does not open or a case does not appear in a real database, drop the claim.
In 2026, ChatGPT, Claude, and Gemini can all browse the web and cite live pages. This dramatically reduces hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → on current facts because the model is reading real sources instead of recalling patterns. Turn this on for anything factual:
Always click the links it shows. A cited source is only useful if it actually says what the model claims.
This prompt helps:
> "Answer the question. Then list which specific claims you are confident about and which you are unsure about or may be guessing."
It is not perfect, but it often surfaces the shaky parts.
If ChatGPT and Gemini independently give the same answer, your confidence can go up. If they disagree, you have found exactly the spot to investigate.
Knowledge check
1. In the 2023 Steven Schwartz case, what was fundamentally wrong with the six court cases cited in his legal brief?
2. According to the lesson, what is the single core thing a large language model actually does?
3. The lesson compares an LLM to a 'brilliant improv actor' asked about a 1990s Bulgarian sci-fi film. What point does this analogy illustrate?
4. Select ALL correct answers. Based on the lesson, which statements accurately describe why hallucinations occur?
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers. Which of the following correctly characterize a hallucination as defined in the lesson?
Sélectionnez toutes les réponses correctes.
If you ever build with the AI tools (through their developer interface, called an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →, a way for programs to send requests to the model), you can cut hallucinations by giving the model the source text yourself. This is called grounding: instead of asking the model to recall, you hand it the facts and tell it to answer only from those.
Here is the idea in a short Python snippet using the OpenAI APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →:
from openai import OpenAI
client = OpenAI()
source_text = """
Our return policy: customers can return items within 30 days
for a full refund. Electronics must be returned within 14 days.
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": (
"Answer ONLY using the provided policy text. "
"If the answer is not in the text, say 'Not specified.'"
),
},
{"role": "user", "content": source_text + "\nHow long do I have to return a laptop?"},
],
)
print(response.choices[0].message.content)The instruction "if the answer is not in the text, say 'Not specified'" gives the model permission to admit it does not know. That single line prevents a lot of confident guessing. The model should answer "14 days" here, and refuse to invent a policy you never gave it.
You do not need to code to use this idea. Even in the chat window, pasting in the actual document and saying "answer only from this text" works the same way.
A quick clarification, because people often blame the wrong thing.
A hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → is not a bug that will get patched away next month. It is a side effect of how these models work. Newer models in 2026 hallucinate less, especially when searching the web, but no model is at zero. Build your habits as if it can always happen, because it can.
It is also not the model being malicious. There is no intent. It is pattern-completion that landed on a confident falsehood.
Schwartz's real mistake was not using ChatGPT. Plenty of lawyers use AI well. His mistake was trusting confident output on a high-stakes, highly verifiable task without checking a single citation.
One search on a legal database would have caught it in minutes. The tool was fine. The missing habit was verification.
That is the whole lesson: the AI is a fast, fluent, sometimes-wrong assistant. Your judgment is the safety check.