Clearing the safety, regulatory, and liability bar for clinical AI

Most cleared AI reaches market through the 510(k) pathway, where the manufacturer shows the device is "substantially equivalent" to an existing legally marketed one. Higher-risk, novel tools may go through De Novo or the more demanding premarket approval (PMA).

Practical CMIO checks:

• Get the clearance letter and the intended use statement. A model cleared to "aid detection of large vessel occlusion on CT" is not cleared to triage every stroke. Using it off its intended use shifts risk onto you.
• Know if it is even a device. Many EHR-embedded predictive tools are marketed as clinical decision supportdecision supportTechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.View full definition → that is exempt from clearance. Exempt does not mean safe. It means you own more of the validation.
• Watch for adaptive models. A model that keeps learning after deployment can drift outside its cleared parameters. The FDA addresses this through Predetermined Change Control Plans (PCCPs), where a manufacturer pre-specifies what changes are allowed without new review.

The FDA maintains a public, searchable list of AI-enabled medical devices it has authorized. Check any vendor claim against it.

🎬 [VIDEO: "What is Software as a Medical Device (SaMD)?" — youtube.com — a short primer on how regulators classify clinical software]

Bar 2: Algorithmic bias audits

A model cleared by the FDA can still fail your patients. Clearance is tested on the manufacturer's data, not yours.

The most cited real-world example: a widely used population health algorithm was found to systematically underestimate the care needs of Black patients because it used historical healthcare spending as a proxy for illness, and less had historically been spent on those patients. The model was not "biased" in code. It faithfully learned a biased reality. (See the original Science study by Obermeyer et al..)

That is the trap: bias usually enters through the label and the data, not the algorithm.

What a bias audit actually looks like

You do not need to be a data scientist to demand these steps. You need to require them and read the results.

1. Define subgroups that matter clinically. Age, sex, race and ethnicity, primary language, insurance type, and disease severity.

2. Measure performance per subgroup, not just overall. An overall accuracy of 92 percent can hide 70 percent accuracy in one group.

3. Pick fairness metrics deliberately. Equal false negative rates matter most when a miss is deadly (missing sepsis). Equal false positive rates matter when a false alarm causes harm (unnecessary biopsy).

A simplified check your data team can run:

If one group's missed rate is far higher, you have a safety problem, whatever the overall number says.

4. Validate on your own population before go-live. This is called local validation or a silent trial: the model runs in the background, its outputs logged but not shown to clinicians, so you can compare predictions to real outcomes without risking patients.

5. Re-audit on a schedule. Populations, coding practices, and care patterns shift. A model that was fair in 2024 can drift. This is performance drift, and it is the single most neglected step.

Bar 3: Malpractice and liability exposure

Here is the question that keeps CMIOs up at night: when the model is wrong and a patient is harmed, who pays?

The current legal reality (and this is general information, not legal advice) is that liability tends to land on the clinician and the institution, not the software vendor. Vendors often classify their tools as decision supportdecision supportTechnologies and processes that turn raw data into actionable insights via reporting, dashboards and analysis, so teams can decide based on facts rather than intuition.View full definition →, keeping the human "in the loop" and therefore responsible.

That creates two opposite risks:

• Automation bias: clinicians over-trust the model and stop thinking. If a physician follows a wrong AI recommendation, "the computer said so" is not a defense.
• Rejection risk: a clinician overrides a correct AI alert and the patient is harmed. Now the existence of the ignored alert can become evidence against them.

You cannot eliminate this tension. You govern it.

Practical liability controls

• Document the intended use and keep clinicians inside it. Deploy the stroke model only for its cleared indication.
• Log every alert, override, and outcome. Your defense in a lawsuit is a paper trail showing a reasonable, monitored process.
• Write override guidance. Tell clinicians when they may disagree with the model and how to record why. Reasoned overrides protect everyone.
• Review your vendor contract for indemnification. Who is liable if the model itself is defective? Involve legal and risk management before signing.
• Set a decommissioning trigger. Define in advance the performance threshold at which you pull a model. A tool with no off-switch is a liability with no ceiling.

Assembling the governance framework

Put the three bars into one standing process. A workable structure for a CMIO:

An AI governance committee. Cross functional: clinical leaders, data science, IT security, legal, risk, ethics, and patient representation. It approves models before deployment and reviews them on a cadence.

A model intake checklist. For every proposed tool:

• Regulatory status: cleared SaMD, exempt CDS, or research use only?
• Intended use statement and its limits.
• Vendor-provided subgroup performance data.
• Plan and dataset for local validation.
• Monitoring plan, drift thresholds, and decommissioning trigger.
• Liability and indemnification review.

A model registry. A living inventory of every AI tool in clinical use, its version, its owner, its last audit date, and its performance. If you cannot list your models, you cannot govern them.

A monitoring dashboard. Ongoing subgroup performance, alert volumes, and override rates. Rising overrides often signal a model losing clinician trust or drifting.

This mirrors emerging consensus frameworks. The Coalition for Health AI (CHAI) publishes assurance standards and a model card format worth using as a template rather than building from scratch.

A quick worked example

Your emergency department wants an AI tool that reads chest X rays and flags likely pneumonia.

1. Regulatory: Confirm it is FDA cleared as SaMD and that "flag suspected pneumonia on adult chest X ray" is its actual intended use. It is not cleared for pediatrics, so you block that use.

2. Bias: Vendor data looks strong overall. You run a silent trial for eight weeks and find lower sensitivity in patients with existing lung scarring. You add a clinician confirmation step for that group.

3. Liability: Radiologists keep final read authority. Every AI flag and every override is logged. Contract review confirms the vendor stands behind defects in the cleared function.

4. Deploy and monitor: The model enters your registry with a quarterly re-audit and a defined sensitivity floor that triggers shutdown.

That is what clearing all three bars looks like in practice.

Key takeaways

• Clearance is the floor, not the finish line. FDA authorization tests a vendor's data, not your patients. Always run local validation before go-live.
• Bias lives in the data and the labels, not usually the algorithm. Audit performance per clinical subgroup, choose fairness metrics based on which error is more harmful, and re-audit for drift.
• Liability lands on the clinician and institution. Keep humans meaningfully in the loop, log every alert and override, and write clear override guidance to defend reasoned decisions.
• Govern with a standing committee, an intake checklist, and a model registry. If you cannot list and monitor every AI tool in clinical use, you do not yet control it.
• Every model needs an off-switch. Define the performance threshold that triggers decommissioning before you deploy, not after harm occurs.