In early 2024, Air Canada lost a small-claims case that should terrify every CDO piloting generative AI. A grieving passenger asked the airline's website chatbot about bereavement fares. The bot confidently invented a policy that didn't exist, told the customer he could claim a refund retroactively, and the tribunal held the airline liable for what its bot said. The chatbot had almost certainly passed its internal demo. It answered fluently. It sounded helpful. It was wrong in a way no one had tested for.
This is the central deception of LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → systems: fluency masquerades as correctness. A traditional model that misclassifies a transaction fails loudly and measurably. An LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions. fails *persuasively*. It produces a confident, grammatical, plausible answer that is subtly or catastrophically false—and it does so non-deterministically, so the same prompt can pass on Tuesday and fail on Thursday.
You already know how to stand up MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → for a classifier: version the model, monitor drift, retrain on a schedule. That playbook does not transfer cleanly. LLMOpsLLMOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → is a different discipline because the failure modes are different, the inputs are unbounded natural language, and the "ground truth" is often a matter of judgment. The hard part of your GenAI program is not building the pilot. It's proving the pilot can be trusted—and building the machinery that keeps proving it after launch. That machinery is what this lesson makes concrete.
The instinct of most teams is to obsess over model selection and prompt engineeringprompt engineeringPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.View full definition →. That's the visible, fun work. But the reason 80% of GenAI pilots stall before production is that nobody built a credible way to answer the question your board will ask: *"How do we know it works?"*
Consider what "works" even means for a retrieval-augmented support assistant. There are at least four distinct dimensions, and they trade off against each other:
A single accuracy number cannot capture this. And unlike a classifier, you cannot compute these dimensions with a simple label comparison, because there is no single correct string. "Your refund will process in 5–7 business days" and "Expect your refund within a week" are both correct.
This is why the discipline splits evaluation into two regimes, and a CDO must fund both:
Offline evaluation runs against a curated, versioned eval set before anything ships. Online evaluation measures behavior against real, unpredictable production traffic. Teams that only do offline evaluation get blindsided by the queries they never imagined. Teams that only monitor online are flying without a pre-flight check. You need both, and they use different tooling.
The single highest-leverage artifact in your entire GenAI program is not the model. It's the golden evaluation set: a curated collection of representative inputs paired with expected behaviors, graded criteria, and known edge cases. This is a data asset you own, that competitors can't buy, and that appreciates over time.
A serious eval set is not 20 questions an engineer typed on a Friday. It should be assembled deliberately:
1. Mine real intent. Pull actual historical queries—support tickets, search logs, agent transcripts. If you're pre-launch, have subject-matter experts write them. Stratify by frequency so common cases are weighted properly.
2. Deliberately over-sample the tail. The failures that create legal and reputational risk live in the edge cases: adversarial prompts, ambiguous questions, out-of-scope requests ("Can I get medical advice?"), and prompt-injection attempts.
3. Attach graded expectations, not just answers. For each item, specify what a good response must contain, must not contain, and how it should behave (e.g., "must cite policy source," "must refuse and escalate").
4. Version it like code. The eval set lives in your repository, is reviewed on change, and every model or prompt release is scored against a pinned version so you can compare apples to apples.
A practical target: 200–500 curated items for a first production pilot, growing continuously as production surfaces new failure modes. Every real-world failure becomes a new permanent test case. That is how the asset compounds.
The obvious problem: who grades 500 open-ended responses every time you tweak a prompt? Human review doesn't scale to the iteration speed you need. The now-standard answer is LLM-as-judge—using a strong model to score outputs against your criteria.
This works, but only if you treat the judge with the same skepticism as any other measurement instrument. The judge itself has biases: it favors longer answers, it prefers responses that match its own style, and it can be gamed. The discipline is to calibrate the judge against human labels. Have your SMEs manually grade a sample of a few hundred outputs, then measure how well your LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → judge agrees with them. If judge–human agreement is high, you can trust it to scale. If it's low, your judge prompt needs work before you rely on it.
A judge prompt should force a rubric-based, structured decision rather than a vague quality vibe:
You are evaluating a support answer for GROUNDEDNESS.
Context provided to the assistant:
{retrieved_context}
User question: {question}
Assistant answer: {answer}
Score groundedness 1-5:
5 = every claim is directly supported by the context
1 = answer contains claims not present in the context
Return JSON: {"score": int, "unsupported_claims": [..], "reasoning": "..."}Notice this forces the judge to *enumerate* unsupported claims. That structure both improves accuracy and gives your team debuggable output—you can see *why* something scored low, not just that it did.
Evaluation tells you how the system behaves in aggregate. Guardrails are the real-time controls that constrain any single response before it reaches a user or acts on a system. Think of evaluation as your quality process and guardrails as your circuit breakers.
Guardrails operate at two points, and mature systems use both:
Input guardrails screen what goes into the model: detecting prompt injection ("ignore your instructions and..."), filtering PII before it hits a third-party APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →, and classifying whether a request is even in scope. An out-of-scope medical question to a banking bot should be caught here, not answered.
Output guardrails screen what comes back: checking for hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → against retrieved sources, blocking toxic or non-compliant content, verifying the response format, and—critically for regulated industries—enforcing that certain answers include disclaimers or trigger human escalation.
The architectural decision a CDO must make is where responsibility sits. Do not let application engineers hand-roll a regex filter and call it governance. Guardrails should be a shared, centrally-owned service layer, so that your policy on PII handling or medical-advice refusal is defined once and enforced consistently across every GenAI application in the enterprise. This is the point where your existing governance mandate becomes executable code. The policy your governance council writes must mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → to a guardrail someone can enforce and audit.
Here's the judgment call that trips teams up: guardrails add latency and cost, because each check is often another model call. An aggressive output-validation guardrail can double your response time. You will make explicit trade-offs—running lightweight checks on every request and expensive verification only on high-risk intents. That risk-tiering is a CDO decision, not an engineering afterthought.
Knowledge check
1. The lesson describes 'the central deception of LLM systems' as fluency masquerading as correctness. Why does this make LLM failures more dangerous than traditional classifier failures?
2. Why does the lesson argue that the standard MLOps playbook (version the model, monitor drift, retrain on schedule) 'does not transfer cleanly' to LLM systems?
3. According to the lesson, what is the primary reason most GenAI pilots stall before reaching production?
4. Select ALL correct answers. Based on the lesson, which characteristics make evaluating an LLM system fundamentally harder than evaluating a traditional classifier?
Select all the correct answers.
5. Select ALL correct answers. What lessons should a CDO draw from the Air Canada chatbot case as illustrated in the excerpt?
Select all the correct answers.
The most dangerous assumption in GenAI is that passing offline evaluation earns you permanent trust. It doesn't. LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → systems degrade in ways classifiers don't. Your vendor silently updates the underlying model. User behavior shifts and starts sending queries your eval set never covered. Your own retrieval corpus goes stale as documents change. A system that scored 92% at launch can quietly drift to 70% with no code change on your side.
So production monitoring is not optional telemetry—it is the mechanism by which trust is *maintained*. A CDO should insist the monitoring stack captures four layers:
The operational discipline that ties it together is a feedback loop: every flagged production failure is triaged, and the genuine failures are promoted into the golden eval set. This is the flywheel. Your eval set gets richer precisely where reality proved you wrong, and the next release is tested against the exact failure that embarrassed you last time. A GenAI program without this loop is not a product; it's a demo on borrowed time.
Concretely, here is the cadence to stand up before a pilot goes to production:
The organizations winning at enterprise GenAI are not the ones with the best models—everyone rents the same frontier models. They win because they built the evaluation and monitoring machinery that lets them iterate fast *and safely*, shipping improvements weekly with confidence instead of shipping a pilot once and praying.
1. Fund the golden eval set before the pilot. It is your most durable, compounding data asset—200–500 curated, versioned cases that over-sample the risky tail. No eval set, no production. Treat it as a board-level artifact, not an engineering side task.
2. Never trust an unvalidated judge. LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition →-as-judge scales grading, but calibrate it against human labels first and force rubric-based, structured scoring. An uncalibrated judge is a confident liar measuring another confident liar.
3. Make guardrails a shared service, not per-app plumbing. Centralize input and output controls so your governance policy is enforced once and audited everywhere, and explicitly risk-tier the expensive checks against latency and cost.
4. Instrument trust as a runtime property. Monitor system, quality, guardrail, and user-feedback layers continuously; a launch-day score means nothing three weeks later once the vendor updates the model or traffic shifts.
5. Close the loop or lose the war. Every real production failure must become a permanent eval case. This flywheel—not model choice—is what separates a defensible GenAI product from a demo living on borrowed time.