Why your model is wrong in ways you won't see

A junior associate at a mid-size firm asks a legal AI assistant to summarize the indemnification clauses across 40 vendor contracts. The tool returns a clean, confident summary. No fabricated case citations, no hallucinated statutes, nothing that would trip a hallucinationhallucinationEine Hallucination liegt vor, wenn ein KI-Modell Output erzeugt, der flüssig und selbstbewusst klingt, aber faktisch falsch, erfunden oder nicht durch die Quelldaten gedeckt ist.Vollständige Definition ansehen → check. It is simply wrong: it systematically misclassifies "mutual indemnification" clauses as "one-way" because the vendor's underlying model was fine-tuned mostly on one-way indemnification templates common in its training set. Nobody catches it until a partner spots the pattern three months later, across dozens of client memos.

This is not hallucinationhallucinationEine Hallucination liegt vor, wenn ein KI-Modell Output erzeugt, der flüssig und selbstbewusst klingt, aber faktisch falsch, erfunden oder nicht durch die Quelldaten gedeckt ist.Vollständige Definition ansehen →. It is bias, drift, and overconfidence, three distinct failure modes that hide behind fluent, well-formatted output. They deserve their own vocabulary and their own checks.

HallucinationHallucinationEine Hallucination liegt vor, wenn ein KI-Modell Output erzeugt, der flüssig und selbstbewusst klingt, aber faktisch falsch, erfunden oder nicht durch die Quelldaten gedeckt ist.Vollständige Definition ansehen → vs. the failures you don't see

HallucinationHallucinationEine Hallucination liegt vor, wenn ein KI-Modell Output erzeugt, der flüssig und selbstbewusst klingt, aber faktisch falsch, erfunden oder nicht durch die Quelldaten gedeckt ist.Vollständige Definition ansehen → gets the headlines: a model invents a case citation or a statute that does not exist. It is loud and checkable. You run the citation through Westlaw or a court database and it either exists or it doesn't.

The three risks in this lesson are quieter:

• Bias: systematic skew from unrepresentative or historically skewed training data.
• Drift: silent degradation or behavior change after a vendor updates the underlying model.
• Confident wrongness: an answer that is factually or analytically incorrect but delivered with the same fluent certainty as a correct one, with no hedging cue for the reader.

None of these trigger a "this looks made up" reflex in the reader, because the output is plausible, well-formatted, and internally consistent. That is precisely what makes them dangerous in a law firm setting, where the finished product (a memo, a contract redline, a due diligence summary) is often trusted on the strength of its tone.

Bias baked into training data

Large language models (LLMs, AI systems trained on huge text corpora to predict and generate language) learn patterns from whatever text they were trained on: public filings, case law, contracts, web text. Legal training data has known skews.

Concrete examples relevant to law firms:

• Jurisdictional skew: most public training data over-represents US federal case law and English-language common law systems. A model asked about Delaware corporate law will likely outperform the same model asked about Luxembourg or Brazilian corporate law, without disclosing the confidence gap.
• Precedent-era skew: older, more heavily cited case law is overrepresented relative to recent rulings, so a model can under-weight a 2024 appellate reversal that overturned decades of precedent.
• Contract-template skew: vendor fine-tuningfine-tuningFine-Tuning passt ein vortrainiertes Modell an eine bestimmte Aufgabe oder Domäne an, indem das Training auf einem kleineren, gezielten Datensatz fortgesetzt wird. Das verbessert Genauigkeit und Stil für den jeweiligen Anwendungsfall.Vollständige Definition ansehen → datasets for contract review tools often lean on templates from a handful of large law firms or deal types (e.g., US tech M&A), which quietly shapes what the model treats as "standard" clause language.

This matters for governance because bias is not a bug you patch once. It is a property of the training distribution, and it resurfaces whenever the tool is pushed outside the domain it was implicitly optimized for.

Regulatory angle: the EU AI Act (Regulation (EU) 2024/1689, in force since August 2024) classifies certain AI uses in the administration of justice as "high-risk," which triggers requirements around data governancedata governanceData Governance umfasst die Regeln, Rollen und Prozesse, die sicherstellen, dass Daten in einer Organisation korrekt, sicher, klar definiert und verantwortungsvoll genutzt werden.Vollständige Definition ansehen →, documentation, and human oversight (see the European Commission's AI Act overview). Legal research and case-outcome-prediction tools sit close to this line, and firms using them should ask vendors for documentation on training data composition, not just accuracy claims.

Silent drift after vendor updates

Most law firms do not train their own models. They license access to GPT, Claude, Gemini, or a legal-specific wrapper (Harvey, CoCounsel, Lexis+ AI, and similar). That means the firm has no control over, and often no visibility into, when the underlying model changes.

Model drift here means: the vendor pushes a new model version or fine-tune, and behavior shifts, sometimes for the better on benchmarks, sometimes worse on your specific use case, with no notification tailored to legal workflows.

A documented industry pattern (widely discussed since 2023, including in OpenAI's and Anthropic's own model cards) is that newer model versions can change formatting habits, verbosity, or even reasoning style in ways that break downstream prompts and templates that were calibrated against the old version. A firm's carefully tuned prompt for extracting termination clauses may silently produce different output structure after a point release, without any error message.

Why this is a governance problem, not just an IT annoyance: if a firm has validated a workflow (say, first-pass NDA review) against version A of a model, and the vendor swaps in version B, the validation is stale. Nobody re-tests it unless someone is explicitly tasked with change management for AI tools, meaning a process to detect, log, and re-validate after upstream model changes.

Practical check: maintain a small, fixed "golden set" of past matters with known correct answers (e.g., 20 contracts with pre-tagged clause types). Re-run this golden set against the tool monthly or after any known vendor update, and compare outputs. This is the AI equivalent of a regression test in software engineering.

# simplified drift check logic
golden_set = load_labeled_contracts()  # 20 contracts, human-verified answers
current_outputs = run_model(golden_set, model_version="current")
baseline_outputs = load_saved_outputs(model_version="baseline")

diff_rate = compare(current_outputs, baseline_outputs)
if diff_rate > threshold:
    flag_for_human_review()

This is not a technical build project for most firms. It is a discipline: keep the golden set, keep the baseline outputs, re-run periodically.

Confident-sounding wrong answers

LLMs are trained to produce fluent, grammatically confident text regardless of their internal certainty. There is no built-in "I'm only 55% sure" signal in a standard chat response, unless the vendor has explicitly engineered one.

This creates a specific law firm risk: the tone of the answer does not correlate with its accuracy. A model can be as confidently wrong about a novel choice-of-law question as it is confidently right about a well-settled point of contract interpretation, and the reader has no cue to tell the difference.

Detection methods differ from hallucinationhallucinationEine Hallucination liegt vor, wenn ein KI-Modell Output erzeugt, der flüssig und selbstbewusst klingt, aber faktisch falsch, erfunden oder nicht durch die Quelldaten gedeckt ist.Vollständige Definition ansehen → checks:

• Calibration testing: periodically ask the model questions where you already know the answer and where the answer is genuinely uncertain or contested, then check whether its expressed confidence (if any) tracks actual correctness.
• Second-model cross-checking: run the same query through a second, independently trained model and flag disagreements for human review, since independent models are unlikely to share the same blind spots.
• Forced uncertainty disclosure: prompt engineeringprompt engineeringPrompt Engineering ist die Praxis, Texteingaben so zu gestalten und zu verfeinern, dass große Sprachmodelle präzise, relevante und verlässliche Ergebnisse liefern.Vollständige Definition ansehen → or vendor features that require the model to cite its source passage or flag jurisdictional assumptions, turning an unfalsifiable confident answer into a checkable one.

The NIST AI Risk Management Framework (a voluntary US framework, first published January 2023) explicitly separates "valid and reliable" from "explainable and interpretable" as distinct trustworthiness characteristics, precisely because fluent output can be reliable-looking without being explainable or correct.

Building this into firm governance

None of these three risks are solved by the same fix. That is the core lesson for governance design:

A firm-level AI governance committee (increasingly common at AmLaw 100 firms as of 2025-2026, though structures vary and no single body regulates this in the US) should own all three checks, not just a generic "AI accuracy" review. The American Bar Association's Formal Opinion 512 (July 2024) on generative AI use already puts the duty of competence and supervision squarely on lawyers, meaning these checks are not optional risk-management theater, they mapmapEinsatz von Software, um wiederkehrende Marketingaufgaben und Kampagnen zu automatisieren und Personalisierung in großem Maßstab über Kanäle wie E-Mail, Web und Social zu ermöglichen.Vollständige Definition ansehen → to existing professional responsibility rules (see the ABA's opinion summary).

Key Takeaways

• Bias, drift, and confident wrongness are distinct from hallucinationhallucinationEine Hallucination liegt vor, wenn ein KI-Modell Output erzeugt, der flüssig und selbstbewusst klingt, aber faktisch falsch, erfunden oder nicht durch die Quelldaten gedeckt ist.Vollständige Definition ansehen →: they produce plausible, well-formatted, wrong output that standard citation-checking will not catch.
• Bias comes from training data composition (jurisdiction, era, template source) and requires domain-specific accuracy testing, not general benchmarks.
• Drift happens silently when vendors update underlying models; maintain a "golden set" of labeled matters and re-test after known updates.
• Confident tone is not a reliability signal in LLMLLMEin Large Language Model ist ein KI-System, das auf riesigen Textmengen trainiert wurde, um Sprache vorherzusagen und zu erzeugen. Damit sind Aufgaben wie Schreiben, Zusammenfassen und Beantworten von Fragen möglich.Vollständige Definition ansehen → output; use calibration testing, cross-model checks, and mandatory source citation to expose it.
• These risks mapmapEinsatz von Software, um wiederkehrende Marketingaufgaben und Kampagnen zu automatisieren und Personalisierung in großem Maßstab über Kanäle wie E-Mail, Web und Social zu ermöglichen.Vollständige Definition ansehen → directly to existing professional duties (ABA Formal Opinion 512) and to emerging regulation (EU AI Act high-risk categories), so governance checks are a compliance matter, not just a technical nicety.