# Measuring outcomes and running post-deployment evaluation
An emergency department deploys an AI sepsis early-warning tool. Six months later, nurses have quietly started ignoring its alerts. Nobody flagged it. The model still fires, dashboards still glow green, and the vendor invoice still gets paid. The AI is technically "live" and clinically dead.
This is the most common failure mode in hospital AI: not a dramatic error, but a slow decay that no one is watching for. This lesson shows you how to build the monitoring and evaluation system that catches it.
In most industries, if a model degrades you lose money. In a hospital, you can harm a patient. That raises the bar for continuous evaluation and it is increasingly a regulatory expectation.
In the US, the FDA (Food and Drug Administration) regulates many AI tools as SaMD (Software as a Medical Device). For adaptive models it promotes a PCCP (Predetermined Change Control Plan), essentially a pre-approved plan for how a model may update and how you will monitor it. In Europe, the EU AI Act (in force since 2024, with high-risk obligations phasing in through 2026 and 2027) classifies most clinical decision AI as "high-risk," requiring post-market monitoring and human oversight.
Translation: continuous evaluation is not a nice-to-have. It is the law's assumption.
A good hospital AI dashboard answers three questions in one glance:
1. Is it working clinically? (outcomes)
2. Are people using it? (adoption)
3. Is the model still valid? (drift)
Skip any one and you get the sepsis story above.
Track the outcome the AI was bought to change, not just model accuracy. Accuracy is an input. Patient outcome is the point.
Example: a sepsis alert tool. Do not just report "AUROC 0.85." (AUROC = Area Under the Receiver Operating Characteristic curve, a 0-to-1 score of how well the model separates sick from not-sick; 0.5 is a coin flip, 1.0 is perfect.) Report:
Always compare against a baseline. Best practice: a concurrent control group or a clean pre-deployment period. Without a comparison, a falling mortality rate could be seasonal, not your AI.
An AI that clinicians override or ignore delivers zero value regardless of its accuracy.
Key metrics:
Alert fatigue is the silent killer of clinical AI. If a nurse gets 40 alerts a shift and 35 are noise, they will tune out the 5 that matter. A rising override rate is often the first sign your tool is on its way to being ignored.
Drift means the world changed but the model did not. Two flavors:
A concrete trigger: your hospital swaps its EHR (Electronic Health Record) vendor. Field mappings change overnight. Models trained on the old feed can silently start receiving garbage.
Here is a minimal drift check comparing the live input distribution to the training distribution:
from scipy.stats import ks_2samp
# training vs last 30 days of live data, one feature (e.g. patient age)
stat, p_value = ks_2samp(train_feature, live_feature)
# Kolmogorov-Smirnov test: small p_value = distributions differ = possible drift
if p_value < 0.05:
alert_ml_team("Data drift detected on feature: age")Run this per feature, nightly. It is cheap insurance. For a fuller open-source treatment of drift monitoring, the Evidently AI documentation is a solid free starting point.
Do not go from demo to hospital-wide. Use staged gates, and define the exit criteria for each stage *before* you start.
The model runs on live data but its outputs are hidden from clinicians. You compare what it *would* have done against what actually happened.
Goal: confirm technical performance on *your* patients. A model trained in Boston may underperform in a rural clinic with different demographics. This is where you catch that cheaply, with zero patient risk.
Exit criterion example: AUROC within 0.05 of vendor's claim on your data, over 60 days.
One unit, real alerts, close human oversight. Now you measure adoption and workflow fit, which shadow mode cannot show.
Watch for: alert volume, override rate, and clinician feedback. A model can be accurate and still fail here because it fires at the wrong moment in the workflow.
Exit criterion example: acceptance rate above an agreed threshold and no unexplained safety events over 90 days.
Roll out unit by unit, not all at once. Keep the dashboard running. Each new unit is effectively a mini-validation because populations differ across departments.
🎬 [VIDEO: "Monitoring Machine Learning Models in Production" - youtube.com - a clear practical walkthrough of drift detection and production ML monitoring concepts that mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.Voir la définition complète → directly onto clinical deployments]
You do not need finance theory here. You need to know whether the clinical benefit exceeds the total cost of running the AI.
Worked example (illustrative figures, not a benchmark):
A sepsis tool costs, all-in, an estimated 200,000 USD per year (license, integration, ML monitoring staff time). Suppose your evaluation credibly attributes a reduction of 15 sepsis deaths and 30 avoided ICU days per year to the tool.
The point is not the exact number. It is the discipline: tie the spend to a measured, attributed outcome, and refuse to count benefits your evaluation cannot actually support. "Attributed" is the hard word. If you cannot show the AI caused the improvement (via control group or pre/post with confounders addressed), you cannot claim the ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète →.
Vérification des acquis
1. The lesson describes a sepsis tool that is 'technically live and clinically dead.' What core concept does this scenario illustrate?
2. Why does the lesson argue that post-deployment evaluation carries a higher bar in healthcare than in most other industries?
3. The lesson insists on tracking patient outcomes rather than only model accuracy metrics like AUROC. What is the conceptual reasoning behind this?
4. Select ALL correct answers about what a good hospital AI monitoring dashboard must track and why.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about the regulatory context for post-deployment evaluation of clinical AI.
Sélectionnez toutes les réponses correctes.
A dashboard with no owner is decoration. Assign responsibility explicitly.
Set thresholds that trigger automatic action, not just an email:
This mirrors the "post-market surveillance" logic the EU AI Act and FDA both expect. You are building an audit trail that proves you were watching.
Sometimes the right answer is to switch the AI off. A tool whose acceptance rate has collapsed, or whose drift cannot be economically fixed, is a liability. Deciding to retire an underperforming model is a sign of a mature program, not a failed one. Build that option into your governance from day one.