# Model risk: when your AI is confidently wrong
A safety monitor at a mid-size biotech once flagged a spike in "mild" adverse events and cleared it automatically for two weeks. The AI triage model was doing exactly what it was trained to do. It just hadn't been trained on the new dosing formulation rolled out that quarter. Nobody caught it until a human reviewer noticed the pattern by hand. The model wasn't broken. It was confidently wrong, which is worse.
This is model risk: the possibility that a deployed AI system produces incorrect or misleading outputs, silently, while appearing to function normally. In pharma, where models now touch pharmacovigilance (the science of detecting and monitoring adverse drug reactions), clinical trial matching, and regulatory submissions, this risk sits next to clinical and financial risk in seriousness. It just gets far less boardroom airtime.
Drift is when the real world changes but the model doesn't. A model trained on 2023 adverse event reports assumes the drug formulation, patient population, and reporting patterns from that period. Change the manufacturing process, add a new indication, or expand into a new country's patient population, and the model's assumptions quietly go stale.
There are two flavors worth distinguishing:
Neither announces itself. Accuracy metrics computed on old validation sets keep looking fine long after real-world performance has degraded.
HallucinationHallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition →, a term borrowed from large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → (LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition →) research, means the model generates output that is fluent, specific, and wrong. In adverse event narrative summarization, an might invent a causality link ("patient's rash was likely caused by Drug X") that isn't supported by the source report. In trial patient matching, a model might cite an eligibility criterion that doesn't exist in the protocol.
This is dangerous precisely because the output reads like careful human reasoning. A tired reviewer scanning 200 AI-generated case summaries a day will not always catch a fabricated detail buried in fluent prose.
If the training data underrepresents a population, the model underperforms for that population, silently. A well-documented pattern: clinical trial datasets have historically skewed toward male, white, and younger participants relative to real-world disease burden (see the FDA's guidance on diversity in clinical trials for the regulatory response). A patient-matching model trained on that history will be worse at finding eligible patients from underrepresented groups, and worse at flagging adverse events that present differently across demographics.
Bias doesn't look like an error message. It looks like a slightly lower match rate for one subgroup that nobody notices because nobody segmented the dashboard that way.
Adverse event (AE) triage. Pharmacovigilance teams use AI to classify incoming case reports by severity and route "serious" cases for expedited regulatory reporting (in the US, expedited reports go to the FDA under 21 CFR 314.80; in the EU, to EudraVigilance, the European Medicines Agency's safety database). A model with drift or bias can misclassify serious cases as routine, delaying a report that regulators expect within 15 calendar days for serious, unexpected reactions. That's not an efficiency problem. That's a compliance and patient safety problem.
Trial patient matching. AI models scan electronic health records to find patients matching trial eligibility criteria. A model trained mostly on structured data from academic medical centers may perform poorly on community hospital records with messier documentation, systematically excluding patients who'd otherwise qualify, and skewing trial enrollment away from real-world populations.
Regulators are catching up. The FDA's discussion paper on AI/ML in drug development (2023) and its broader digital health center outline expectations for transparency, monitoring, and human oversight, though comprehensive binding rules specific to pharma AI are, as of early 2026, still evolving rather than fully codified. In the EU, the AI Act (entered into force 2024, with phased obligations through 2027) classifies many health-related AI systems as "high-risk," triggering requirements for risk management systems, data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.View full definition →, human oversight, and post-market monitoring. Pharmacovigilance AI plausibly falls into this high-risk bucket given its link to patient safety.
The practical governance response, regardless of exact regulatory timing, centers on a few concrete checks:
1. Model cards and documentation: a standing record of what data trained the model, its known limitations, and intended use (see Google's original Model Cards proposal for the template most of the industry has converged on).
2. Drift monitoring in production: automated statistical tests comparing live input distributions to training distributions, with alerts, not just quarterly manual review.
3. Human-in-the-loop sign-off: no AE triage classification or trial match should reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → a regulatory action or patient contact without a qualified human able to override it.
4. Subgroup performance audits: accuracy, sensitivity, and false-negative rates reported by demographic subgroup, not just in aggregate.
A simplified drift check, the kind a data science team would actually run monthly, looks like this:
from scipy.stats import ks_2samp
# Compare distribution of a key feature (e.g., patient age)
# between training data and last month's production data
statistic, p_value = ks_2samp(training_ages, production_ages_last_month)
if p_value < 0.01:
print("Significant distribution shift detected: investigate before trusting outputs")This is the Kolmogorov-Smirnov test, a standard statistical method for comparing two distributions. It won't catch hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → or bias on its own, but it's a cheap first tripwire for drift.
Knowledge check
1. Why is a model being 'confidently wrong' considered worse than a model that simply breaks or crashes?
2. A pharmacovigilance model was trained on adverse event data before a new dosing formulation was introduced. It keeps clearing new adverse event reports as 'mild' without adjustment. Which type of model risk does this best illustrate?
3. Why might a model experiencing drift continue to show good performance on standard validation metrics?
4. Select ALL correct answers about the distinction between data drift and concept drift.
Select all the correct answers.
5. Select ALL correct answers about why model risk deserves serious attention in pharma settings.
Select all the correct answers.
Before any AI model touches AE triage or patient matching in production, a governance-minded team should be able to answer:
None of this eliminates model risk. The goal is to make failures visible and correctable quickly, rather than silent for two weeks like the biotech case at the top of this lesson.
🎬 [VIDEO: "The dangers of AI hallucinations in healthcare" - youtube.com - a short explainer on how generative AI produces plausible but false clinical content, and why domain experts must stay in the loop]