# Model risk in clinical and lab settings
In 2021, researchers testing dozens of COVID-19 chest imaging models found that not a single one was clinically usable. Many had learned to detect the hospital, not the disease: a model trained where sick patients were scanned lying down had simply learned that "lying down equals COVID." Deployed elsewhere, it broke. This is model risk, and in medicine it does not just cost money. It misdiagnoses people.
Model risk is the risk that an AI system produces wrong or harmful outputs, and that you act on them. In banking the fallout is financial. In biotech and medtech the fallout is a missed tumor, a wrong drug dose, or a failed clinical trial that burns years of pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → value.
Three failure modes cause most real-world damage. Learn to name them.
Dataset shift happens when the data a model sees in production differs from its training data. The world moves; the model does not.
Flavors you will meet:
The Epic Sepsis Model is the canonical warning. An external validation published in *JAMA Internal Medicine* (2021) found it performed far worse than the vendor advertised, missing most sepsis cases at the alert threshold used, while flooding clinicians with alerts. See the study summary via PubMed.
If your training data underrepresents a group, the model underperforms for that group, quietly.
The most cited example: pulse oximeters and skin tone. Studies (including a widely referenced 2020 *NEJM* letter) showed devices overestimated blood oxygen in Black patients, hiding dangerously low levels. The pattern generalizes to AI: a dermatology classifier trained mostly on light skin will miss melanoma on dark skin.
Cohort bias also strikes drug discovery. A model trained on genomic data skewed toward European-ancestry cohorts (still the majority of public datasets as of 2026) can propose targets that do not translate across populations, wasting downstream trial spend.
The scary one. The model does not crash. It just gets worse, and nobody notices because outputs still look plausible.
Causes: upstream data pipelinedata pipelineETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.Voir la définition complète → changes (a lab renames a field, units switch from mg/dL to mmol/L), seasonal population changes, a firmware update on an instrument. Without monitoring, performance can drift for months before an audit or a bad outcome surfaces it.
Regulators want you to categorize and rate risk before deployment, not after. Use two axes borrowed from clinical risk management (ISO 14971, the standard for medical device risk management) and applied to AI.
Risk score = Severity x Probability x Detectability gap
A radiology triage tool that reprioritizes worklists.
Risk score = 5 x 3 x 5 = 75 out of a possible 125.
Now add drift monitoring with weekly performance checks. Detectability gap drops to 2.
New score = 5 x 3 x 2 = 30.
Same model, same clinical stakes, but the guardrail cut measured risk by 60 percent. This is exactly the argument you make to a regulator or a board: the number moved because of a specific control.
*(This scoring scheme is a teaching tool, not a regulatory-mandated formula.)*
Names you must know for 2026.
United States, FDA (Food and Drug Administration). AI-enabled medical devices are regulated as SaMD (Software as a Medical Device). The FDA has cleared over 1,000 AI/ML-enabled devices to date (FDA's own running list, updated periodically, is the source; treat any single count as an estimate). The key modern instrument is the Predetermined Change Control Plan (PCCP): you pre-declare how your model will update and be monitored, so retraining does not require a brand new submission. The FDA finalized PCCP guidance in December 2024.
Europe. Two layers stack:
The practical takeaway: in Europe a medical AI product must satisfy both the device rules and the AI Act. They overlap but are not identical.
Concrete checks, roughly in order.
External validation. Test on data from sites, scanners, and populations the model never trained on. If performance drops sharply, you have covariate shift baked in.
Subgroup performance. Report accuracy sliced by age, sex, ancestry, skin tone, device type. A single headline AUC (Area Under the Curve, a measure of classifier accuracy from 0.5 to 1.0) hides the failures that get people hurt.
Calibration, not just accuracy. A model can rank patients correctly but output wrong probabilities. A "90 percent risk" must mean 90 out of 100 such patients actually have the condition.
Drift monitoring in production. This is the fix for silent degradation. Track input distributions and output distributions over time and alert on deviation.
Here is the minimal lens, using a standard statistical test to flag covariate shift on one input feature:
from scipy.stats import ks_2samp
# reference = feature values from training/validation
# live = same feature from the last production window
stat, p_value = ks_2samp(reference_feature, live_feature)
if p_value < 0.01:
alert("Distribution shift detected: investigate before trusting outputs")The Kolmogorov-Smirnov test compares two distributions. A tiny p-value means the live data no longer looks like your reference data. That is your early warning, not a diagnosis.
🎬 [VIDEO: "The Problem with AI in Healthcare" - youtube.com - accessible overview of how medical AI models fail on real-world datareal-world dataRWD, données collectées en dehors des essais cliniques contrôlés : dossiers médicaux, claims d'assurance, données de dispositifs connectés, base des Real-World Evidence (RWE). and why validation matters]
Vérification des acquis
1. A COVID imaging model performed well in its original hospital but broke when deployed elsewhere because it had learned that patients scanned lying down were more likely to have COVID. What does this failure most directly illustrate?
2. A pathology model trained on slides from one scanner brand shows collapsing confidence when it encounters blurrier slides from a newly installed scanner. Which type of dataset shift best describes this?
3. New clinical guidelines mean that the same set of lab values now implies a different level of patient risk than when a model was trained. Why is this especially dangerous compared to a simple change in patient mix?
4. Select ALL correct answers. Why does model risk in clinical/lab settings differ from model risk in banking?
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers. The Epic Sepsis Model's poor external validation results serve as a warning about which concepts?
Sélectionnez toutes les réponses correctes.
A model with no owner is a liability with no address. Premium AI governance in biotech assigns three roles explicitly:
Two more non-negotiables:
Human oversight. A clinician must be able to see, question, and override the model. The EU AI Act mandates this for high-risk systems. Automation bias (humans deferring to the machine) is itself a risk to design against.
Logging and traceability. Every input, output, model version, and override should be recorded. When something goes wrong, and it will, you need to reconstruct what the model saw and decided.