# Running the pre-deployment guardrail checklist
A sepsis prediction model went live at hundreds of US hospitals and, when independently studied in 2021, missed roughly two-thirds of sepsis cases while flooding clinicians with false alarms. The tool had been sold and deployed widely. Nobody ran the checklist you are about to learn.
This lesson gives you the concrete go/no-go gate a hospital should clear before flipping the switch on any AI tool, from a radiology triage algorithm to an ambient documentation assistant.
An AI tool in a hospital is a medical device or a clinical decision aid. That means it sits under real oversight.
FDA clearance or a CE mark tells you the vendor cleared a bar. It does not tell you the tool works on *your* patients, *your* workflows, *your* data. The pre-deployment checklist is your local safety gate.
Useful reference: the FDA maintains a public list of AI-enabled medical devices
Treat each item as pass/fail. One fail means no-go until fixed.
Bias here means the model performs unevenly across patient groups, producing worse care for some.
The classic case: a widely used care-management algorithm, studied in *Science* in 2019, used past healthcare spending as a proxy for health need. Because less money was historically spent on Black patients at equal sickness, the algorithm systematically under-referred them. Same accuracy overall, badly unequal by race.
Your audit must slice performance by:
Compute the same metrics per subgroup, not just in aggregate. A simple check:
# Sensitivity (recall) per demographic subgroup
for group, df in patients.groupby("race_ethnicity"):
tp = ((df.pred == 1) & (df.actual == 1)).sum()
fn = ((df.pred == 0) & (df.actual == 1)).sum()
sensitivity = tp / (tp + fn)
print(f"{group}: sensitivity = {sensitivity:.2f}, n = {len(df)}")Go/no-go rule: define a maximum acceptable gap before you run this, not after. For example, sensitivity must not vary by more than a preset threshold across major subgroups. If one group has too few patients to measure, that itself is a finding.
Shadow mode means running the AI silently on live cases: it makes predictions, but clinicians never see them and care is unaffected. You compare its outputs against what actually happened.
This is the single most important step, and the one most often skipped under vendor pressure.
Why local data matters: a pneumonia detector trained on one health system's chest X-rays can degrade badly on another's, because scanner models, patient populations, and labeling habits differ. This is distribution shift: the world your model sees in production differs from the world it was trained on.
Run shadow mode long enough to capture:
Worked example. Suppose in a three-month shadow run your sepsis tool flags 400 alerts. Of those, 120 are true sepsis cases. There were 150 actual sepsis cases total.
Now decide: is a 70% false-alarm rate survivable for your nurses, or will it cause alert fatigue (staff ignoring alarms because most are noise)? These numbers are illustrative, but the calculation is exactly the one your governance committee should demand.
No clinical AI should act autonomously without a human in the loop. The question is *how* the override works in practice.
Check for:
A subtle failure mode is automation bias: humans defer to the machine even when it is wrong. Ambient AI scribes that draft clinical notes are a live 2026 example. The physician is supposed to review and sign, but if the draft looks fluent, hallucinated details can slip through. Your design must make review real, not theater.
🎬 [VIDEO: "The danger of AI in health care" - youtube.com - a clinician-focused talk on why oversight and validation matter more than model accuracy alone]
Passing the gate at launch is not the finish line. Models drift. Populations change. A tool that was safe in January can be unsafe by July.
Before deployment, define the numbers you will watch and the levels that trigger action:
Assign an owner. A metric nobody watches is not a guardrail.
Vérification des acquis
1. A hospital acquires a diagnostic AI tool that has received FDA clearance. Why is the pre-deployment guardrail checklist still necessary before going live?
2. The care-management algorithm that used past healthcare spending as a proxy for health need illustrates which core concept?
3. Under the described checklist, how should a hospital treat a single failed item?
4. Select ALL correct answers about the regulatory landscape for clinical AI described in the lesson.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers describing why the sepsis model example is a cautionary tale for pre-deployment review.
Sélectionnez toutes les réponses correctes.
A rollback trigger is a pre-agreed condition that shuts the tool off automatically or forces an urgent review, decided *before* launch so nobody has to argue during a crisis.
Define them concretely:
That last one matters. Vendors ship silent model updates. A version that passed your gate is not the version running six months later. Contractually require notification and re-validation rights before any model change. If the vendor cannot commit to that, treat it as a no-go signal.
Have a tested manual fallback. If the AI triage tool goes dark, staff must know exactly what the pre-AI process was and be able to run it today.
Assemble a small cross-functional group to own sign-off: a clinician who uses the tool, a data or informatics lead, someone from compliance or risk, and a patient-safety representative. Each checklist item gets a documented pass, fail, or conditional pass with a fix date.
This documentation is not bureaucracy. Under the EU AI Act, high-risk systems require a risk management file and post-market monitoring. In the US, the same records protect you in an adverse-event review. The checklist doubles as your audit trail.
Keep the bar simple: no local validation, no deployment. A vendor's FDA clearance or CE mark is a starting condition, not a substitute for proving the tool works on your patients.