The governance landscape for AI in biotech and medtech

2. HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation. (US health data privacy)

The Health Insurance Portability and Accountability Act (HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation.) governs Protected Health Information (PHI): identifiable patient data. Your model ingests PHI every 15 minutes. That means:

• A Business Associate Agreement (BAA) with any vendor touching the data.
• Minimum-necessary data use and audit logging.
• Rules on whether PHI can be used to retrain the model (often it cannot, without de-identification or authorization).

3. EU AI Act (high-risk classification)

If this hospital operates in the EU or serves EU patients, the EU AI Act (Regulation 2024/1689, phasing in through 2026 and 2027) applies. AI used as a medical device, or as a safety component of one, is classified high-risk. High-risk obligations include:

• A risk management system across the model lifecycle.
• Data governanceData governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.Voir la définition complète → and bias checks on training data.
• Human oversight ("human-in-the-loop").
• Technical documentation and post-market monitoring.
• Registration in an EU database.

Note the layering: the same tool is a medical device under EU rules (MDR) *and* high-risk under the AI Act. You satisfy both, not one.

4. GxP (good practice quality standards)

GxP is shorthand for "good practice" quality regulations (GLP, GCP, GMP) used across pharma and diagnostics. The relevant one here is often GAMP 5 (Good Automated Manufacturing Practice), the framework for validating computerized systems. Its core principle: validation. You must document that the system does what you claim, reproducibly, with 21 CFR Part 11 controls on electronic records and signatures (audit trails, access control, data integrity).

Why this is hard: the regimes do not talk to each other

Each regime has a different owner, vocabulary, and trigger. FDA cares about *intended use*. HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation. cares about *data*. The AI Act cares about *risk tier and oversight*. GxP cares about *validated, auditable systems*.

The failure mode is not one big gap. It is small orphaned obligations that no single team feels responsible for.

| Question | Who often assumes it is not theirs |

|---|---|

| Is retraining on PHI allowed? | Data science thinks Legal; Legal thinks IT |

| Who signs the FDA PCCP? | Regulatory Affairs vs. clinical lead |

| Who monitors model drift post-launch? | Everyone assumes someone else |

| Where is the validation evidence stored? | Quality vs. vendor |

Building the governance operating model

The fix is an explicit RACI-style ownership map (Responsible, Accountable, Consulted, Informed) tied to each regime. A workable structure:

AI Governance Committee (accountable). Cross-functional: Regulatory Affairs, Quality, Legal/Privacy, Clinical, Data Science, IT Security. It owns the go/no-go decision.

Named owners per obligation:

• Regulatory Affairs owns FDA classification, submission, and the PCCP.
• Privacy Officer owns HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation., the BAA, and de-identification rules.
• Quality owns GxP validation and 21 CFR Part 11 audit trails.
• AI Act lead (often Regulatory or a dedicated compliance role) owns high-risk documentation, bias testing, and human-oversight design.
• Clinical lead owns intended-use definition and the human-in-the-loop workflow.
• MLOps/Data Science owns drift monitoring and model cards.

The single most useful artifact is a model registry entry that binds all of this to one model version.

If any field is blank or "unknown," you do not deploy.

The AI-specific risks the checklist must catch

Regulations are the floor. The model risks are what actually harm patients.

Dataset shift / drift. The model was trained on one hospital's population. A new patient mix, a changed lab vendor, or a coding-practice change degrades accuracy silently. This is the classic failure of deployed clinical AI: performance looks fine at launch, then quietly decays.

Bias across subgroups. Aggregate accuracy hides subgroup gaps. A sepsis model can perform worse on, say, patients with darker skin tones if oxygen-saturation inputs are systematically biased, a documented issue with pulse oximetry. The AI Act explicitly requires this check.

Automation bias. Clinicians over-trust the score and stop thinking. Human oversight only works if the workflow makes overriding easy and expected.

Alert fatigue. Too many false positives and staff ignore all alerts, including the true ones. This is a real, measured harm, not a UX nitpick.

🎬 [VIDEO: "Real-world evidence for AI in healthcare" - youtube.com - overview of how deployed clinical AI models drift and why post-market monitoring matters]

Pre-deployment guardrails: the go-live checklist

Before the model touches a live patient, run these checks. Each maps to a regime and an owner.

1. Intended-use statement locked and matched to FDA classification. (Regulatory)

2. Validation dossier complete: performance on a held-out set that reflects *this* hospital, plus subgroup breakdowns. (Quality + Data Science)

3. Bias report across age, sex, race, and clinically relevant subgroups. (Data Science, per AI Act)

4. PHI handling signed off: BAA in place, retraining rules explicit, audit trail on. (Privacy + Quality)

5. Human oversight designed and tested: can a clinician see the reasoning and override without friction? (Clinical)

6. Monitoring plan live: drift metric, threshold, alert routing, and a rollback plan *before* launch. (MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.Voir la définition complète →)

7. Change control: how updates get approved, tied to the PCCP if adaptive. (Regulatory + Quality)

A worked example: is a drift alert warranted?

Suppose validation AUROC (Area Under the ROC Curve, a 0-to-1 accuracy measure) was 0.82. Your governance rule: investigate if it drops more than 0.05.

Three months in, monitoring reports AUROC 0.76.

Drop = 0.82, 0.76 = 0.06, which exceeds the 0.05 threshold.

Action: trigger review, notify the named MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.Voir la définition complète → owner, and consider pausing per the rollback plan. No debate about *whether* to act, because the threshold and owner were defined pre-launch. That is governance doing its job.

(These numbers are illustrative, not from a specific product.)

The direction of travel for 2026

Expect convergence pressure, not simplification. The FDA continues expanding its AI/ML device framework and PCCP use. The EU AI Act's high-risk obligations are landing in stages through 2026-2027. Regulators increasingly expect lifecycle governance, not a one-time approval. The competitive edgecompetitive edgeA lasting edge over competitors: a resource, capability or position they cannot easily replicate, letting a firm earn above-average returns over time.Voir la définition complète → goes to organizations that treat governance as an operating capability, not a legal formality.

Key Takeaways

• One clinical AI model triggers four regimes at once (FDA, HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation., EU AI Act high-risk, GxP). Compliance means satisfying all of them, not choosing one.
• The real risk is orphaned obligations. Fix it with a named owner per obligation and a single model-registry entry binding everything to one model version.
• Regulations are the floor; drift, bias, automation bias, and alert fatigue are what harm patients. Your go-live checklist must test for each.
• Define drift thresholds, rollback plans, and owners before launch, so response is automatic, not a debate. A 0.06 AUROC drop past a 0.05 rule triggers action, no meeting required.
• Governance is a lifecycle capability in 2026, not a one-time approval. Build the operating model once and reuse it across every model you deploy.

*This lesson is educational and not legal, medical, or regulatory advice.*