The governance landscape for AI in biotech and medtech

A 600-bed academic hospital flips the switch on a sepsis-prediction model. It scans electronic health records every 15 minutes and flags patients likely to deteriorate. Clinically, it looks like a win. Legally, it just triggered four overlapping regulatory regimes at once. Miss any one of them and the deployment is not just risky, it is potentially illegal.

This lesson maps that landscape and shows you how to build a governance operating model that assigns clear ownership before go-live.

The four regimes hitting one model

Our sepsis model is not governed by "AI regulation." It is governed by a stack of regimes that were mostly written for other purposes and now apply simultaneously.

1. FDA (US medical device rules)

If software is intended to diagnose, treat, or predict disease, the US Food and Drug Administration (FDA) may regulate it as Software as a Medical Device (SaMD): software that performs a medical function without being part of a hardware device.

A sepsis-prediction tool that drives clinical decisions likely needs FDA clearance or approval. The FDA has authorized hundreds of AI-enabled devices already; it publishes the running list of AI/ML-enabled medical devices publicly. The FDA's key concern with AI is the locked vs. adaptive question: a model that retrains itself in the field changes behavior, so the FDA introduced the Predetermined Change Control Plan (PCCP), a pre-authorized description of how a model may update without a new submission.

2. HIPAA (US health data privacy)

The Health Insurance Portability and Accountability Act (HIPAA) governs Protected Health Information (PHI): identifiable patient data. Your model ingests PHI every 15 minutes. That means:

• A Business Associate Agreement (BAA) with any vendor touching the data.
• Minimum-necessary data use and audit logging.
• Rules on whether PHI can be used to retrain the model (often it cannot, without de-identification or authorization).

3. EU AI Act (high-risk classification)

If this hospital operates in the EU or serves EU patients, the EU AI Act (Regulation 2024/1689, phasing in through 2026 and 2027) applies. AI used as a medical device, or as a safety component of one, is classified high-risk. High-risk obligations include:

• A risk management system across the model lifecycle.
• Data governanceData governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.View full definition → and bias checks on training data.
• Human oversight ("human-in-the-loop").
• Technical documentation and post-market monitoring.
• Registration in an EU database.

Note the layering: the same tool is a medical device under EU rules (MDR) *and* high-risk under the AI Act. You satisfy both, not one.

4. GxP (good practice quality standards)

GxP is shorthand for "good practice" quality regulations (GLP, GCP, GMP) used across pharma and diagnostics. The relevant one here is often GAMP 5 (Good Automated Manufacturing Practice), the framework for validating computerized systems. Its core principle: validation. You must document that the system does what you claim, reproducibly, with 21 CFR Part 11 controls on electronic records and signatures (audit trails, access control, data integrity).

Why this is hard: the regimes do not talk to each other

Each regime has a different owner, vocabulary, and trigger. FDA cares about *intended use*. HIPAA cares about *data*. The AI Act cares about *risk tier and oversight*. GxP cares about *validated, auditable systems*.

The failure mode is not one big gap. It is small orphaned obligations that no single team feels responsible for.

Building the governance operating model

The fix is an explicit RACI-style ownership map (Responsible, Accountable, Consulted, Informed) tied to each regime. A workable structure:

AI Governance Committee (accountable). Cross-functional: Regulatory Affairs, Quality, Legal/Privacy, Clinical, Data Science, IT Security. It owns the go/no-go decision.

Named owners per obligation:

• Regulatory Affairs owns FDA classification, submission, and the PCCP.
• Privacy Officer owns HIPAA, the BAA, and de-identification rules.
• Quality owns GxP validation and 21 CFR Part 11 audit trails.
• AI Act lead (often Regulatory or a dedicated compliance role) owns high-risk documentation, bias testing, and human-oversight design.
• Clinical lead owns intended-use definition and the human-in-the-loop workflow.
• MLOps/Data Science owns drift monitoring and model cards.

The single most useful artifact is a model registry entry that binds all of this to one model version.

model: sepsis_predictor
version: 2.3.0
intended_use: "Early sepsis risk flag, adult inpatient, decision-support only"
fda_status: 510k_cleared        # or: exempt / pending
fda_pccp: true                  # adaptive updates pre-authorized
data: PHI (HIPAA); BAA_signed: true; retrain_on_PHI: false
eu_ai_act_class: high_risk
human_oversight: clinician_confirms_before_action
gxp_validation: GAMP5_complete; part11_audit_trail: enabled
owners:
  regulatory: name
  privacy: name
  quality: name
monitoring:
  drift_metric: AUROC
  alert_threshold: 0.05_drop
  review_cadence: monthly

If any field is blank or "unknown," you do not deploy.

The AI-specific risks the checklist must catch

Regulations are the floor. The model risks are what actually harm patients.

Dataset shift / drift. The model was trained on one hospital's population. A new patient mix, a changed lab vendor, or a coding-practice change degrades accuracy silently. This is the classic failure of deployed clinical AI: performance looks fine at launch, then quietly decays.

Bias across subgroups. Aggregate accuracy hides subgroup gaps. A sepsis model can perform worse on, say, patients with darker skin tones if oxygen-saturation inputs are systematically biased, a documented issue with pulse oximetry. The AI Act explicitly requires this check.

Automation bias. Clinicians over-trust the score and stop thinking. Human oversight only works if the workflow makes overriding easy and expected.

Alert fatigue. Too many false positives and staff ignore all alerts, including the true ones. This is a real, measured harm, not a UX nitpick.

🎬 [VIDEO: "Real-world evidence for AI in healthcare" - youtube.com - overview of how deployed clinical AI models drift and why post-market monitoring matters]

Pre-deployment guardrailsguardrailsRules and controls that keep an AI system inside safe, legal and on-brand boundaries, blocking outputs and actions that cross the line.View full definition →: the go-live checklist

Before the model touches a live patient, run these checks. Each maps to a regime and an owner.

1. Intended-use statement locked and matched to FDA classification. (Regulatory)
2. Validation dossier complete: performance on a held-out set that reflects *this* hospital, plus subgroup breakdowns. (Quality + Data Science)
3. Bias report across age, sex, race, and clinically relevant subgroups. (Data Science, per AI Act)
4. PHI handling signed off: BAA in place, retraining rules explicit, audit trail on. (Privacy + Quality)
5. Human oversight designed and tested: can a clinician see the reasoning and override without friction? (Clinical)
6. Monitoring plan live: drift metric, threshold, alert routing, and a rollback plan *before* launch. (MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition →)
7. Change control: how updates get approved, tied to the PCCP if adaptive. (Regulatory + Quality)

A worked example: is a drift alert warranted?

Suppose validation AUROC (Area Under the ROC Curve, a 0-to-1 accuracy measure) was 0.82. Your governance rule: investigate if it drops more than 0.05.

Three months in, monitoring reports AUROC 0.76.

Drop = 0.82, 0.76 = 0.06, which exceeds the 0.05 threshold.

Action: trigger review, notify the named MLOps owner, and consider pausing per the rollback plan. No debate about *whether* to act, because the threshold and owner were defined pre-launch. That is governance doing its job.

(These numbers are illustrative, not from a specific product.)

The direction of travel for 2026

Expect convergence pressure, not simplification. The FDA continues expanding its AI/ML device framework and PCCP use. The EU AI Act's high-risk obligations are landing in stages through 2026-2027. Regulators increasingly expect lifecycle governance, not a one-time approval. The competitive edgecompetitive edgeA lasting edge over competitors: a resource, capability or position they cannot easily replicate, letting a firm earn above-average returns over time.View full definition → goes to organizations that treat governance as an operating capability, not a legal formality.

Key Takeaways

• One clinical AI model triggers four regimes at once (FDA, HIPAA, EU AI Act high-risk, GxP). Compliance means satisfying all of them, not choosing one.
• The real risk is orphaned obligations. Fix it with a named owner per obligation and a single model-registry entry binding everything to one model version.
• Regulations are the floor; drift, bias, automation bias, and alert fatigue are what harm patients. Your go-live checklist must test for each.
• Define drift thresholds, rollback plans, and owners before launch, so response is automatic, not a debate. A 0.06 AUROC drop past a 0.05 rule triggers action, no meeting required.
• Governance is a lifecycle capability in 2026, not a one-time approval. Build the operating model once and reuse it across every model you deploy.

*This lesson is educational and not legal, medical, or regulatory advice.*