# When AI helps and when it does not
A lawyer in 2023 asked ChatGPT to find legal precedents for a court filing. It gave him six perfect-looking cases, complete with citations. The judge later discovered that all six were invented. The lawyer was fined and made national news.
The AI did exactly what it was built to do: produce fluent, confident text. The problem was the task. He needed verified facts. He got a convincing draft.
This lesson gives you a simple rule for telling those two situations apart, so you never make that mistake.
A large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → (the technology behind ChatGPT, Claude, and Gemini) predicts likely-sounding text. It does not "know" things the way a database does.
That means it is brilliant at producing plausible language and unreliable at guaranteeing exact truth. When plausible and true happen to line up, great. When they don't, you get a confident wrong answer, also known as a "hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition →" (the AI making something up while sounding sure).
So the question is never "Is AI smart enough?" The question is "Does this task reward plausible, or does it demand verified?"
Before you trust an AI's output, ask two things:
1. Is this a draft or a final answer?
A draft is something you will review and edit. A final answer goes straight out the door with your name on it.
2. Is it easily verifiable?
Can you check it in seconds, or would a mistake slip through unnoticed?
Here is the rule:
These play to the model's strengths: language, structure, and first drafts.
You give it the content. It just changes the form. Low risk, because the facts came from you.
Reformat these rough notes into a polite client email.
Do not add any facts I didn't provide:
- project delayed 1 week
- waiting on their logo files
- new delivery date March 12A blank page is expensive. A rough draft you can fix is cheap.
If a brainstorm idea is slightly off, you just don't use it. No harm done.
These need exact facts, current data, or real accountability. The lawyer's mistake lives here.
The model may produce a number that looks right and is wrong by billions.
A model's core knowledge has a "training cutoff" (the date its built-in knowledge stops). On its own, it does not know today's news, today's prices, or what happened this morning.
In 2026, most tools (ChatGPT, Gemini, Claude) can search the web when you ask, which helps a lot. But "can search" is not "did search," and search results can still be misread. Always check the actual source link.
If a human is responsible for it being correct, a human must verify it.
Models have improved, but they still slip on multi-step arithmetic. For real calculations, use a calculator or a spreadsheet, or ask the AI to write code that does the math instead of doing it in its head.
Most real work lives here. The trick is making verification fast.
A great move: ask the AI to show its sources so you can check them.
Summarize the main causes of the 2008 financial crisis.
For each cause, give one specific source I can look up to verify it.Then you actually click the sources. If it can't produce a real, checkable source, treat the claim as unverified.
Another tactic: ask it to flag its own uncertainty.
Answer my question below. At the end, list any part of your
answer you are NOT confident about or could not verify.
Question: What are the visa requirements for a US citizen
traveling to Brazil in 2026?This won't catch everything, but it surfaces the riskiest claims so you know where to look.
For a clear, free primer on how these systems generate text (and why they make things up), see Google's Introduction to Large Language Models.
Why Large Language Models Hallucinate
Knowledge check
1. According to the lesson, what is the fundamental reason a language model can produce a confident but wrong answer?
2. The lesson says the right question to ask is never 'Is AI smart enough?' What should you ask instead?
3. A colleague uses AI to write a customer-facing legal disclaimer that will be published unedited, and the accuracy is hard to check. Under the two-question rule, what light is this?
4. Select ALL tasks that qualify as 'green light' (safe to let AI run) according to the lesson.
Select all the correct answers.
5. Select ALL statements that correctly describe the two questions in the decision rule.
Select all the correct answers.
Let's combine everything. Say you need to email a customer about a refund policy.
Step 1 (Green): Draft the structure.
Draft a friendly email to a customer asking about our refund window.
Leave a [PLACEHOLDER] anywhere a specific policy detail should go.
Don't invent any policy numbers.Notice the instruction: don't invent details. You force the AI to leave gaps instead of guessing. The gaps are honest. A fabricated "30-day policy" is dangerous.
Step 2 (Red, so you do it): Fill in the facts.
You look up your actual refund policy and replace each [PLACEHOLDER] yourself. This is the part with real accountability, so a human owns it.
Step 3 (Green): Polish.
Here's my completed email with real policy details.
Tighten the wording and keep the tone warm. Don't change any numbers.The AI did the heavy lifting on language. You owned the facts. That division of labor is the whole game.
Watch for the sneaky shift. A task starts as a harmless draft and slides into a final answer without you noticing.
You ask for a "draft" report, like the numbers, paste them into a real document, and ship it. Now those AI-generated numbers are a final, accountable answer, and nobody verified them.
The fix: decide up front which parts are draftable language and which parts are facts you must verify. Mark the facts. Never let an unchecked number graduate to "final."
These actions are compiled in the role's Playbook.
Before you hit send on anything AI touched:
1. Did I add or verify every specific fact, number, name, and date?
2. Did I click any source links the AI gave me?
3. If this needed current info, did the tool actually search, and did I check the result?
4. Would I be comfortable if a mistake here had my name on it?
If any answer is "no," you're not done.
[PLACEHOLDER] markers instead of guessing.