In 2021, researchers documented a phenomenon that keeps radiology AI teams awake: a chest X-ray model trained at one hospital lost accuracy when the hospital swapped its scanner vendor. The pixels looked the same to a human. To the model, the world had shifted. Nothing failed loudly. Sensitivity just quietly eroded, one scan at a time.
This is the core problem of AI in medtech. A drug, once approved, does not change its chemistry. An AI model degrades the moment reality drifts away from its training data. That is why regulators treat deployed AI as a living system that needs continuous watching, not a product you ship and forget.
This lesson builds the post-market surveillance loop: drift detection, audit trails, rollback, and the regulatory reports you file when things go wrong.
Why models drift after deployment
"Drift" means the statistical relationship the model learned no longer holds. Two flavors matter:
Data drift: the inputs change. New scanner, new patient population, a software update that alters image contrast.
Concept drift: the relationship between inputs and the correct answer changes. A new disease variant, a revised diagnostic guideline.
Concrete example: a sepsis prediction model deployed across US hospitals was found in a 2021 external validation (published in JAMA Internal Medicine) to perform far worse than its marketing claimed, partly because local populations differed from the development set. Deployment context is not a detail. It is the whole game.
The regulatory frame you are operating in
You cannot design monitoring without knowing what you must report. Name the real bodies:
US FDA regulates AI-enabled medical devices as SaMD (Software as a Medical Device). The FDA maintains a public list of AI-enabled medical devices, which passed 1,000 authorized devices (FDA figure, as of late 2024). Post-market obligations run through MDR (Medical Device Reporting): serious adverse events and device malfunctions must be reported, generally within 30 days.
The FDA's Predetermined Change Control Plan (PCCP) framework (guidance finalized 2024) lets you pre-authorize specific model updates so you can retrain without a new submission, if you defined the update protocol up front.
Europe: AI medical devices fall under the MDR (Medical Device Regulation 2017/745) plus the EU AI Act (in force 2024, high-risk obligations phasing in through 2026 and 2027). Most diagnostic AI counts as high-risk, triggering mandatory post-market monitoring, logging, and human oversight requirements. Notified Bodies (private conformity assessors, for example TÜV SÜD) audit your quality system.
The practical takeaway: continuous monitoring is not best practice. It is a legal obligation in both markets.
Building the surveillance loop
Think of four connected stages: detect, investigate, act, report.
1. Detect: drift alerts
You monitor two things: the inputs and the outputs.
Input monitoring compares live data distributions to your training baseline. A common statistical tool is Population Stability Index (PSI), which measures how much a distribution has moved.
For a radiology model, you might track mean pixel intensity, image resolution metadata, and the mix of scanner vendors. If a new scanner pushes PSI above 0.25, you get an alert before accuracy visibly drops.
Output monitoring watches the model's own behavior: prediction confidence distribution, positive rate, and, where you have ground truth, actual performance. If your model normally flags 8% of chest scans as abnormal and suddenly flags 20%, that is a signal even before pathology confirms anything.
The hard part in medicine: ground truth is delayed. You often do not know the model was wrong until a biopsy, a follow-up scan, or a missed diagnosis surfaces weeks later. So you lean on proxy signals (confidence, input drift, radiologist override rates) as early warnings.
2. Investigate: the audit trail
When an alert fires, you need to reconstruct exactly what happened. Your audit trail should capture, for every prediction:
Model version and identifier
Input hash and key metadata (device, timestamp, site)
The output and confidence score
Whether a human agreed or overrode it
The EU AI Act explicitly requires automatic logging for high-risk systems, retained for the device lifetime. This is not optional bureaucracy. When a clinician asks "why did the AI miss this tumor," you must be able to answer with the exact model version and inputs. If you cannot reproduce a decision, you cannot defend it and you cannot fix it.
Radiologist override rate is one of the richest signals. If overrides jump from 5% to 15% in one clinic, that clinic is seeing something your model handles poorly. Log it, segment by site, and watch it.
3. Act: rollback protocols
When you confirm a problem, you need a fast, rehearsed response. Borrow from software engineering:
Version everything. Every deployed model has an immutable version tag. You never overwrite; you deploy alongside.
Rollback = repoint traffic to the previous known-good version. This should take minutes, not a re-submission cycle.
Canary deployment: release a new model to a small slice of traffic (say one hospital, or 5% of scans) and compare its behavior to the incumbent before full rollout.
Human-in-the-loop fallback: if confidence collapses, route those cases to a radiologist rather than auto-reporting. A degraded AI that defers is safer than one that guesses.
Define your decommission trigger in advance: at what measured performance drop do you pull the model entirely? Write it down before the crisis, not during it.
Knowledge check
1. Why does the lesson argue that AI models require continuous post-market surveillance while an approved drug does not?
2. A hospital swaps its scanner vendor, and a chest X-ray model's sensitivity slowly erodes even though images look identical to humans. Which phenomenon best describes this?
3. A model is retrained on the same patients, but a newly emerged disease variant changes what symptoms actually indicate infection. What type of drift is this, and why does it matter?
MULTIPLE CHOICE
4. Select ALL correct answers about why the sepsis prediction model performed worse than advertised in external validation.
Select all the correct answers.
MULTIPLE CHOICE
5. Select ALL correct answers describing components of a sound post-market surveillance loop for deployed medical AI.
Select all the correct answers.
4. Report: regulatory notification
When an AI failure causes or could cause patient harm, reporting is mandatory and time-bound.
US (FDA MDR): manufacturers report deaths and serious injuries, and malfunctions likely to cause harm if they recur, generally within 30 calendar days. Events requiring urgent remedial action have a 5-day window. (FDA timelines, current as of 2025.)
EU (MDR): serious incidents must be reported to the competent authority, generally within 15 days; death or unanticipated serious deterioration triggers a 10-day window; an immediate serious public health threat requires reporting within 2 days. (MDR Article 87 timelines.)
A drifting model that caused a missed diagnosis is reportable. Note the asymmetry: EU incident timelines can be tighter than FDA's, so a global deployment defaults to the strictest clock.
A worked example
Your lung nodule detection model runs across 12 hospitals. Monitoring shows:
Hospital 7 installed a new CT scanner. Input PSI on pixel intensity climbs to 0.31 (above the 0.25 threshold).
Radiologist override rate at Hospital 7 rises from 6% to 14% over two weeks.
One confirmed missed nodule, later caught on follow-up.
Response sequence:
1. Alert fires on PSI breach. Investigate via audit trail, isolate to Hospital 7 and the new scanner.
2. Act: route Hospital 7 scans to human-primary review (AI as second reader only). Rollback is not needed elsewhere; the drift is localized.
3. Report: the missed nodule is a serious incident. Under EU MDR, file within 15 days. Under FDA MDR, within 30.
4. Remediate: if your PCCP covers scanner-vendor retraining, retrain and validate on the new scanner data under the pre-authorized protocol.
Localized drift, contained fast, reported honestly. That is the loop working.
Common failure modes
Monitoring only accuracy. By the time accuracy drops with confirmed ground truth, patients were already affected. Watch inputs and proxies.
No versioning. You cannot roll back what you cannot name.
Silent auto-updates. An unlogged model change is a compliance and safety hole.
Ignoring subgroups. A model can be stable overall while degrading badly for one hospital, one demographic, or one device.
Key takeaways
AI devices degrade; monitor them as living systems. Drift (data or concept) is the default outcome, not the exception. Both FDA and EU AI Act mandate post-market surveillance.
Watch inputs and proxies, not just accuracy. Use PSI on input distributions, prediction rates, and radiologist override rates as early warnings, because ground truth arrives late in medicine.
Log everything, immutably. Model version, inputs, outputs, human overrides. The EU AI Act requires lifetime logging, and you cannot fix or defend what you cannot reconstruct.
Rehearse rollback before the incident. Version tags, canary releases, human-in-the-loop fallback, and a pre-written decommission trigger turn a crisis into a procedure.
Know your reporting clocks. EU MDR serious incidents generally within 15 days (2 days for public health threats); FDA MDR generally 30 days (5 for urgent). Global deployments default to the strictest deadline.