The pre-deployment checklist that regulators expect

None of these frameworks certify "the algorithm." They certify the system around the algorithm: data lineagedata lineageData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.View full definition →, testing, monitoring, and accountability.

The checklist: validation

Validation answers one question: can you prove the model does what you claim, on the population you claim, under the conditions you claim?

1. Intended use statement. Written, specific, and narrow. "Flags potential drug-drug interactions for adult inpatients on formulary X" is auditable. "Improves patient safety" is not.

2. Training and validation data provenance. Where did the data come from, what population does it represent, and does that population match your deployment site? A sepsis-prediction model trained on one US academic hospital's data can fail silently at a rural EU clinic with different demographics and lab equipment. This is the single most common finding in post-market AI failures.

3. Performance metrics against a predefined threshold. Sensitivity, specificity, calibration, not just headline accuracy. Set the acceptance threshold *before* you see the test results, not after.

4. Subgroup performance. Break results out by age, sex, ethnicity, and site. A model that is 95% accurate overall but degrades sharply for one subgroup is a fairness and safety liability, and both FDA and EMA now expect this analysis by default.

5. External or prospective validation. Retrospective validation on historical data is a floor, not a ceiling. Regulators increasingly ask for a prospective silent run (the model runs live but its output is hidden from clinicians) before go-live.

6. Change control plan. If the model retrains or updates, how will you re-validate without re-submitting from scratch every time? FDA's PCCP mechanism exists precisely so you can pre-specify the boundaries of acceptable model drift.

The checklist: human-in-the-loop sign-off

"Human-in-the-loop" (HITL) means a qualified person reviews or can override the AI output before it affects patient care, and the system is designed so that override is genuinely easy, not theoretical.

7. Defined decision authority. Name the role, not the person: "attending physician" or "qualified pharmacist," not "a human." Document what they can and cannot delegate to the tool.

8. Override capability, tested. Can a clinician actually reject the AI's recommendation in the workflow, in under a few seconds, without a workaround? If overriding takes longer than complying, humans will rubber-stamp the AI. That's "automation bias" and it's a top-cited risk in EMA's reflection paper.

9. Escalation path for edge cases. What happens when the model outputs "uncertain" or a confidence score below threshold? There must be a defined fallback, not silence.

10. Competency and training records. Staff using the tool must be trained on its limitations, and that training must be documented and refreshed. Auditors ask for training logs, not just policies.

11. Alert fatigue monitoring. If clinicians start ignoring 90% of alerts because of false positives, that's a governance failure even if the model's stated accuracy hasn't changed. Track override rates as a live metric, not a one-time check.

The checklist: audit trails

Audit trail requirements come straight from GxP culture: if it isn't documented, it didn't happen.

12. Full logging of inputs, outputs, and model version. Every prediction should be traceable to the exact model version, input data, and timestamp. This is non-negotiable for FDA 21 CFR Part 11 (electronic records) compliance in the US.

13. Human decision logging. Record whether the clinician accepted, modified, or overrode the AI recommendation, and ideally why. This is the dataset you'll need later to prove the human oversight wasn't cosmetic.

14. Data lineage documentation. Where each training and inference data point came from, how it was labeled, and by whom. EMA explicitly flags data provenancedata provenanceData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.View full definition → gaps as a common deficiency.

15. Incident and drift monitoring log. A running record of post-deployment performance checks, flagged anomalies, and corrective actions. This is your evidence of "continued verification," a phrase used in both FDA and EMA guidance.

16. Retention policy aligned to record-type rules. Clinical and pharmacovigilance records typically require multi-year retention (often 10+ years depending on jurisdiction and product type); confirm the applicable period rather than assuming a default.

Here's a minimal example of what a per-prediction audit record might contain in practice:

Nothing exotic here. The point is that every field answers a question an inspector will ask.

Assembling the go/no-go call

Before go-live, three questions decide the outcome:

1. Does validation evidence meet a pre-agreed threshold, on the actual deployment population? If validation was done on different data than the go-live site, that's a "no" until re-validated.

2. Can a named human role override the system in real time, with that override logged? If override exists only on paper, that's a "no."

3. Can you reconstruct any single decision, end to end, six months from now? If logs are incomplete or model versioning is untracked, that's a "no."

A "yes" on all three doesn't guarantee regulatory approval, but its absence guarantees a finding.

🎬 [VIDEO: "FDA's Approach to AI in Medical Devices" - youtube.com - search for FDA Digital Health Center of Excellence talks on AI/ML-based SaMD oversight and the Predetermined Change Control Plan]

Key Takeaways

• Regulators (FDA, EMA) evaluate the governance system around an AI tool, not just model accuracy: validation methodology, human oversight design, and audit trail completeness are the real inspection points.
• Validation must include subgroup performance and, ideally, prospective testing on the actual deployment population, not just retrospective accuracy on training-adjacent data.
• Human-in-the-loop only counts if override is fast, logged, and role-defined; slow or cosmetic override invites automation bias and regulatory findings.
• Every prediction should be traceable: model version, inputs, output, and the human's action, logged and retained per applicable record-keeping rules (e.g., US 21 CFR Part 11).
• Treat the EU AI Act's high-risk classification and FDA's PCCP mechanism as the two frameworks most likely to define your documentation burden through 2026 and beyond.