The lawyer who trusted ChatGPT in court and paid the price

The motion was filed. The citations looked clean. The witnesses had names, statements, apparent credibility. There was only one problem: none of them existed.

A New Mexico defense lawyer, working on a criminal case, used ChatGPT to help build part of the legal record and submitted documents that referenced fabricated testimony from witnesses who were never real. When the court investigated, the lawyer's explanation was direct and, in its own way, revealing: "I didn't know that AI could hallucinate facts." Reported by Ars Technica in 2026, the case resulted in formal punishment from the court.

What actually happened

The specifics matter here, because this was not a case of a distracted professional cutting corners on a minor administrative task. This was testimony, attributed to named individuals, submitted in a criminal defense. The lawyer apparently used ChatGPT to generate or research content, accepted what the model returned at face value, and filed it.

Courts have zero tolerance for invented citations. Lawyers have professional obligations to verify every factual claim they submit. The punishment that followed was a direct consequence of treating a generative AI system as a reliable source of record rather than a drafting assistant that needs supervision.

What makes this case stick is the lawyer's own statement. The admission "I didn't know" is not a defense, but it is an honest one, and it points to a gap that exists across many professions right now. A significant number of people using these tools in 2026 still have an intuitive mental model of AI as something like a very fast search engine: you put a question in, accurate information comes out. That mental model is wrong in a specific and consequential way.

Why it still matters

ChatGPT, Claude, Gemini and comparable models do not retrieve facts. They predict the most statistically plausible next tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → given the input and their training. When that process works well, the output reads like a well-informed answer. When it fails, the output reads exactly the same way. The model has no internal flag that says "I'm uncertain here." It continues generating fluent, confident prose regardless of whether the underlying content is accurate, approximate, or invented.

This is what hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → means in practice. The model is not lying. It is not malfunctioning. It is doing exactly what it was designed to do, predicting plausible text, and in certain conditions that process produces content that is factually false but syntactically and stylistically indistinguishable from content that is true.

Legal work is the clearest example of a domain where that failure mode is catastrophic, but it is far from the only one. Think about medical summaries, financial due diligence memos, compliance documentation, client-facing research reports. Any professional context where a fabricated fact, sourced to a nonexistent study or a made-up precedent, could cause material harm belongs in this category.

The New Mexico case was not the first of its kind. In 2023, New York lawyers Steven Schwartz and Peter LoDuca submitted a ChatGPT-generated brief containing fake case citations and were sanctioned by the court. The fact that a comparable incident occurred again years later, in 2026, with a lawyer who still did not know hallucination was possible, tells you something about how slowly practical AI literacy spreads even as tool adoption accelerates.

There is a structural problem underneath the individual story. AI products are designed to be easy to start using. The friction is minimal. Nothing in the interface warns you that the confident paragraph you just received might contain invented names or nonexistent sources. The risk of confident wrong answers scales with the stakes of the domain, and the tool does not know what domain you are working in.

The takeaway for you

The lesson is not "don't use AI." The lesson is narrower and more actionable: stop treating model output as a terminal step and start treating it as a first draft that requires domain-appropriate verification.

What that looks like varies by context. For a lawyer, it means every name, citation, and factual claim generated by an AI must be checked against primary sources before filing. For a financial analyst, it means every figure, company name, or regulatory reference needs to be cross-referenced with official documentation. For a medical professional, it means clinical detail generated by a model is a starting point for your own lookup, not a conclusion.

A few practical principles that hold up across roles:

• Never use AI output as a primary source. It can help you draft, structure, and explore, but the sourcing burden stays with you.
• Be more skeptical of specific details than of general summaries. Models hallucinate most often on proper nouns: names, dates, case numbers, citations, statistics.
• If the verification step feels tedious, that is a sign the task may not be appropriate for AI-assisted drafting at this stage.
• Build verification into your workflow before you submit, not after someone challenges the document.

One final point worth noting: the New Mexico lawyer's punishment was not primarily about using AI. It was about filing unverified information with a court. The professional obligation to verify predates AI by centuries. What changed is that AI makes it faster and easier to generate plausible-looking unverified content, which means the gap between "drafted" and "verified" is wider than it used to be and matters more.

The tool is not the problem. The assumption that drafting equals verification is. Any professional who closes that gap, by treating AI output as material to be checked rather than conclusions to be filed, will use these tools well. Anyone who does not will eventually have their own version of this story.