# Clearing the safety, regulatory, and liability bar for clinical AI
A sepsis prediction model flags a patient two hours before a nurse would have. The nurse escalates, the patient survives. Same model, different day: it stays silent on a septic patient because the training data underrepresented their demographic, and the delay contributes to a death. Same code. Two very different legal, ethical, and regulatory stories.
As a Chief Medical Information Officer (CMIO), the physician executive who owns clinical technology decisions, you sign off on which of these systems touches real patients. This lesson builds the governance framework you need before any model goes live.
Every clinical AI deployment must pass through three gates. Skip one and you are exposed.
1. Regulatory: Is this software legally cleared to do what you are using it for?
2. Safety and bias: Does it work reliably across your actual patient population?
3. Liability: When it errs, who is accountable, and can you defend the decision to use it?
Let us take them in order.
Software as a Medical Device (SaMD) is the FDA term for software that performs a medical function without being part of a hardware device. A smartphone app that reads a skin lesion image and returns a cancer risk score is SaMD. The imaging monitor it runs on is not.
Most cleared AI reaches market through the 510(k) pathway, where the manufacturer shows the device is "substantially equivalent" to an existing legally marketed one. Higher-risk, novel tools may go through De Novo or the more demanding premarket approval (PMA).
Practical CMIO checks:
The FDA maintains a public, searchable list of AI-enabled medical devices it has authorized. Check any vendor claim against it.
🎬 [VIDEO: "What is Software as a Medical Device (SaMD)?" — youtube.com — a short primer on how regulators classify clinical software]
A model cleared by the FDA can still fail your patients. Clearance is tested on the manufacturer's data, not yours.
The most cited real-world example: a widely used population health algorithm was found to systematically underestimate the care needs of Black patients because it used historical healthcare spending as a proxy for illness, and less had historically been spent on those patients. The model was not "biased" in code. It faithfully learned a biased reality. (See the original Science study by Obermeyer et al..)
That is the trap: bias usually enters through the label and the data, not the algorithm.
You do not need to be a data scientist to demand these steps. You need to require them and read the results.
1. Define subgroups that matter clinically. Age, sex, race and ethnicity, primary language, insurance type, and disease severity.
2. Measure performance per subgroup, not just overall. An overall accuracy of 92 percent can hide 70 percent accuracy in one group.
3. Pick fairness metrics deliberately. Equal false negative rates matter most when a miss is deadly (missing sepsis). Equal false positive rates matter when a false alarm causes harm (unnecessary biopsy).
A simplified check your data team can run:
# Per-subgroup false negative rate for a sepsis model
for group, df in patients.groupby("race_ethnicity"):
fn = ((df.model_flag == 0) & (df.true_sepsis == 1)).sum()
actual_positives = (df.true_sepsis == 1).sum()
fnr = fn / actual_positives
print(f"{group}: missed {fnr:.0%} of true sepsis cases")If one group's missed rate is far higher, you have a safety problem, whatever the overall number says.
4. Validate on your own population before go-live. This is called local validation or a silent trial: the model runs in the background, its outputs logged but not shown to clinicians, so you can compare predictions to real outcomes without risking patients.
5. Re-audit on a schedule. Populations, coding practices, and care patterns shift. A model that was fair in 2024 can drift. This is performance drift, and it is the single most neglected step.
Here is the question that keeps CMIOs up at night: when the model is wrong and a patient is harmed, who pays?
The current legal reality (and this is general information, not legal advice) is that liability tends to land on the clinician and the institution, not the software vendor. Vendors often classify their tools as decision supportdecision supportTechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.Voir la définition complète →, keeping the human "in the loop" and therefore responsible.
That creates two opposite risks:
You cannot eliminate this tension. You govern it.
Vérification des acquis
1. The lesson opens with a sepsis model that saves one patient but stays silent on another whose demographic was underrepresented in training data. What core governance point does this contrast illustrate?
2. According to the lesson, which type of clinical software is the FDA LEAST likely to regulate?
3. A vendor claims their new AI tool is 'substantially equivalent' to an existing legally marketed device. Which FDA pathway are they pursuing?
4. Select ALL correct answers about the three bars every clinical AI deployment must clear.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about Software as a Medical Device (SaMD) as described in the lesson.
Sélectionnez toutes les réponses correctes.
Put the three bars into one standing process. A workable structure for a CMIO:
An AI governance committee. Cross functional: clinical leaders, data science, IT security, legal, risk, ethics, and patient representation. It approves models before deployment and reviews them on a cadence.
A model intake checklist. For every proposed tool:
A model registry. A living inventory of every AI tool in clinical use, its version, its owner, its last audit date, and its performance. If you cannot list your models, you cannot govern them.
A monitoring dashboard. Ongoing subgroup performance, alert volumes, and override rates. Rising overrides often signal a model losing clinician trust or drifting.
This mirrors emerging consensus frameworks. The Coalition for Health AI (CHAI) publishes assurance standards and a model card format worth using as a template rather than building from scratch.
Your emergency department wants an AI tool that reads chest X rays and flags likely pneumonia.
1. Regulatory: Confirm it is FDA cleared as SaMD and that "flag suspected pneumonia on adult chest X ray" is its actual intended use. It is not cleared for pediatrics, so you block that use.
2. Bias: Vendor data looks strong overall. You run a silent trial for eight weeks and find lower sensitivity in patients with existing lung scarring. You add a clinician confirmation step for that group.
3. Liability: Radiologists keep final read authority. Every AI flag and every override is logged. Contract review confirms the vendor stands behind defects in the cleared function.
4. Deploy and monitor: The model enters your registry with a quarterly re-audit and a defined sensitivity floor that triggers shutdown.
That is what clearing all three bars looks like in practice.