# Diagnosing model risk in clinical AI
In 2019, a study published in *Science* found that a widely used US care management algorithm, applied to roughly 200 million people annually, systematically under-referred Black patients to high-risk care programs. The model was not fed race. It used healthcare costs as a proxy for illness. Because less money was historically spent on Black patients at the same level of sickness, the algorithm concluded they were healthier. They were not.
That is model risk. Not a bug in the code, but a mismatch between what the model learned and what the hospital needed it to do. This lesson dissects the four failure modes that hurt hospitals most, using real deployed cases, and the checks that catch them before a patient is harmed.
Model risk is the risk of loss (clinical, financial, or reputational) from a model that is wrong, misused, or misunderstood. In banking this concept is formalized. In healthcare it is younger but converging fast, driven by the US FDA (Food and Drug Administration) and the EU AI Act, which classifies most clinical decision supportdecision supportTechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.View full definition → as "high-risk AI" with mandatory monitoring obligations.
Four failure modes dominate. Learn to name them.
Dataset shift means the data the model sees in production differs from the data it trained on. A model trained on one hospital's patients degrades at another.
The classic case: Epic's Sepsis Model, deployed in hundreds of US hospitals. An external validation at the University of Michigan (published in *JAMA Internal Medicine*, 2021) found it missed roughly two-thirds of sepsis cases and generated frequent false alarms in a population different from Epic's development data. The model was not broken. It was moved.
Three sub-types to watch:
The check: never trust a vendor's internal AUC. Demand local external validation on your own patient data before go-live, and again after any major population change.
> AUC (Area Under the Curve): a score from 0.5 (coin flip) to 1.0 (perfect) measuring how well a model separates sick from healthy patients.
Automation bias is the human tendency to over-trust a machine's output and stop thinking critically. In the ICU, where a nurse manages alarms every few minutes, this is acute.
Two directions of harm:
The governance fix is not better AI. It is workflow design: calibrate alert thresholds to real staffing capacity, log override rates, and treat a rising override rate as a red flag that the model has lost clinical trust.
This one kills quietly. A model performs well at launch, then drifts down over months as practice patterns, equipment, and coding change. Nobody notices because there is no alarm for "the model is slowly getting worse."
A concrete driver: a hospital swaps its lab analyzer vendor. Creatinine values shift slightly in scale. A kidney-risk model calibrated on the old scale now mis-scores every patient, silently.
You do not need deep math to catch decay. Track the model's calibration: do predicted risks match observed outcomes?
Suppose a readmission model flags 1,000 patients per quarter as "high risk (predicted 30%)." At launch, 300 were actually readmitted (30%, well calibrated). Two quarters later:
Predicted high-risk readmission rate: 30%
Flagged patients: 1,000
Model expects readmissions: 300
Actual observed readmissions: 180
Observed / Expected ratio = 180 / 300 = 0.60An O/E ratio of 0.60 means the model now over-predicts risk by 40%. Clinicians are being sent to patients who will not be readmitted, wasting scarce case-management time. That drift is invisible on a dashboard that only shows "model is running." It is obvious on an O/E chart.
Set a trigger: if O/E drifts outside, say, 0.8 to 1.2 for two consecutive periods, pause and revalidate. (Bands are illustrative; set yours with your clinical governance committee.)
Return to the opening case. The harm was not intent. It was a proxy variable (cost) standing in for the real target (health need), combined with historical inequity baked into that proxy.
Where this shows up in hospitals:
The check is subgroup performance analysis. Do not report one AUC. Report it per group: by race, sex, age band, and payer type. A model with 0.85 AUC overall and 0.65 for one subgroup is not an 0.85 model. It is a discriminatory one.
The US Office for Civil Rights and Section 1557 of the Affordable Care Act now explicitly cover discrimination through clinical algorithms. Inequitable AI is a compliance exposure, not just an ethics one.
Knowledge check
1. In the care management algorithm case, the model under-referred Black patients despite never being fed race as an input. What is the core conceptual lesson this illustrates?
2. The lesson defines model risk as loss from a model that is 'wrong, misused, or misunderstood' rather than a bug in the code. Why is this distinction important?
3. The Epic Sepsis Model performed well in development but missed most sepsis cases at the University of Michigan. This best demonstrates which principle about model validation?
4. Select ALL correct answers about dataset shift in clinical AI.
Select all the correct answers.
5. Select ALL correct answers about why regulatory frameworks matter for clinical model risk.
Select all the correct answers.
Governance turns these four risks into a checklist. Anchor it to real frameworks.
Reproduce performance on your own population before go-live. Insist on a model card: a short document stating training population, intended use, known limitations, and subgroup performance.
Decide explicitly whether a human must confirm each output (human-in-the-loop) or can override after the fact (human-on-the-loop). Log override rates as a live safety signal.
Ship monitoring with the model, not later. Track calibration (O/E ratios), alert volume, and input drift. Assign an owner. A model with no owner is an incident waiting for a name.
Not once at procurement. Populations shift. Re-run subgroup analysis on a schedule tied to your governance committee.
Every model needs an off-switch and a documented trigger for using it. The Epic sepsis case shows what happens when a widely deployed model cannot be quickly re-scoped.