# The hospital AI governance operating model that actually works
A sepsis prediction model went live across a large US health system and quietly underperformed for months, firing alerts that clinicians learned to ignore. When independent researchers finally studied a widely deployed sepsis tool, they found it missed most cases and generated large numbers of false alarms. The technology was not the failure. The governance was: no one owned the decision to deploy it, no one monitored it after go-live, and no one had authority to pull it.
This lesson shows you how to build the committee, the sign-off chain, and the checks that stop that story from repeating.
AI in a hospital is not one thing. It spans:
Each carries different risk. An AI that drafts a discharge summary is not the same as one that flags a stroke on a CT scan. A governance model that treats them identically will either strangle the low-risk tools or under-supervise the high-risk ones.
The operating model has one job: route each AI use case to the right level of scrutiny, assign a named owner for each decision, and keep watching after deployment.
Stand up a single AI Governance Committee with clear membership. In most US health systems the core is:
Keep it small enough to decide. A 25-person committee decides nothing. Aim for 8 to 12 voting members with a defined quorum.
Do not make this an IT committee with clinicians invited. Clinical risk is the point. If radiology deploys an imaging AI that changes what a radiologist looks at, the chief of radiology must own that, not a project manager.
Not every use case needs full committee review. Tier your use cases so trivial tools move fast and high-risk tools get scrutiny.
A workable three-tier model:
| Tier | Description | Example | Sign-off required |
|------|-------------|---------|-------------------|
| Low | No direct clinical action, no PHI risk | Meeting transcription, staff scheduling | Department head + IT security |
| Medium | Touches PHI or supports (not drives) clinical work | Ambient scribe, coding assistant | Add CMIO + compliance |
| High | Influences diagnosis, treatment, or triage | Sepsis alert, imaging triage, dosing support | Full committee + clinical validation |
This tiering echoes the risk-based logic in real regulation. The EU AI Act, which entered into force in 2024 with obligations phasing in through 2026 and 2027, classifies most medical AI as high-risk, triggering requirements for risk management, data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.Voir la définition complète →, human oversight, and post-market monitoring. Many US-based medical AI products already meet a version of these through FDA processes.
United States:
Europe:
Governance note: FDA clearance does not mean a tool is safe *in your hospital*. Clearance is based on the manufacturer's data. Your patient mix, your EHR configuration, and your workflow are different. Local validation is your job.
MapMapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.Voir la définition complète → the decision explicitly. For a high-tier clinical model, a clean chain looks like this:
1. Intake and tiering (data science lead + CMIO): confirm the tier and the clinical claim.
2. Security and privacy review (CISO + privacy officer): where does data go, who can access it, is there a signed BAA (Business Associate Agreement, the HIPAAHIPAAHealth Insurance Portability and Accountability Act, loi américaine imposant la protection des données de santé (PHI). Violations : amendes jusqu'à 1,9M$ par catégorie de violation. contract binding a vendor to protect PHI)?
3. Model risk review (data science lead): performance, bias testing, known failure modes.
4. Local clinical validation (relevant department head): test on *your* data before go-live.
5. Committee vote (full committee): go / no-go with conditions.
6. Deployment owner named (usually the department head): one person accountable for the live tool.
No single person should be able to push a high-risk model to patients alone. That is the entire point.
🎬 [VIDEO: "AI in Healthcare: Governance and Trust" — youtube.com — an accessible overview of building trustworthy clinical AI oversight]
Before a high-risk model goes live, run these:
1. Local performance validation. Test the model on a recent slice of *your* patients, not the vendor's benchmark. Report sensitivity and specificity on your population.
2. Bias and subgroup testing. Break performance down by age, race, sex, and payer where legally permitted. A model can look fine overall and fail badly for one group.
Here is the minimal lens every committee should be able to read:
# Subgroup performance check before deployment
import pandas as pd
def subgroup_report(df, group_col, y_true="label", y_pred="prediction"):
out = []
for g, sub in df.groupby(group_col):
tp = ((sub[y_pred]==1) & (sub[y_true]==1)).sum()
fn = ((sub[y_pred]==0) & (sub[y_true]==1)).sum()
fp = ((sub[y_pred]==1) & (sub[y_true]==0)).sum()
sens = tp / (tp + fn) if (tp+fn) else float("nan")
ppv = tp / (tp + fp) if (tp+fp) else float("nan")
out.append({"group": g, "n": len(sub),
"sensitivity": round(sens,3), "ppv": round(ppv,3)})
return pd.DataFrame(out)If sensitivity is 0.82 for one group and 0.61 for another, that gap is a governance decision, not a footnote.
3. Alert burden estimate. For an alerting tool, do a simple calculation. If a sepsis model fires on 8% of 3,000 daily encounters, that is 240 alerts a day. If its positive predictive value is 20%, then 192 of those 240 are false alarms. Clinicians will tune it out. That math must be on the table before go-live.
4. Human oversight design. Define exactly how a clinician can override, ignore, or escalate. AI recommends; a licensed human decides.
5. Post-market monitoring plan. Who reviews live performance, how often, and what triggers a rollback. This is the step the sepsis story skipped.
Vérification des acquis
1. The sepsis prediction model story is presented as a failure of governance rather than technology. What is the core lesson it illustrates?
2. Why does the lesson argue that a hospital needs an AI governance model that differentiates between use cases rather than treating all AI identically?
3. A hospital deploys ambient documentation AI that drafts visit notes and an imaging algorithm that flags strokes on CT scans. According to the operating model's logic, how should these be governed?
4. Select ALL correct answers about the stated purpose of the hospital AI governance operating model.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about the role of the CMIO on the AI Governance Committee.
Sélectionnez toutes les réponses correctes.
Deployment is the start, not the finish. Models degrade when the world changes, called drift. A coding change, a new patient population, or an EHR upgrade can silently break a model.
Set a monitoring cadence:
Assign the deployment owner to bring a one-page dashboard to the committee. Define the rollback authority in advance: the CMIO or department head can suspend a tool immediately, without waiting for the next meeting. Speed matters when patients are involved.
Ambient scribes and chatbots add a new failure mode: hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.Voir la définition complète →, where the model produces fluent but false content. A drafted note that invents a medication is a patient safety event.
Guardrails: require clinician review and sign-off on every AI-drafted note, log the original AI output, and never let generative output post to the record unreviewed. Treat the AI as a very fast intern whose work always gets checked.