Clinical AI explainability: why clinicians ignore alerts

September 11, 2026 · 5 min · Free, with full transcript

Why do clinical decision support tools with strong validation scores get ignored on the ward? The position here is that the blocker is calibration and explainability, not model accuracy. A model that is right 94 percent of the time gives the clinician a lifetime average, not a way to judge the single alert in front of them. Epic's sepsis prediction model, and the external Michigan study that found missed cases and false alarms, is the worked example of how that ends in alert fatigue. You come away with two design rules for clinical AI screens, a way to read OpenAI's medical benchmark claims against O'Reilly Radar adoption data, and a one-question test to run before any go-live.

Listen to the podcast

4 min

Chapters

Key takeaways

  • Show the evidence behind an alert, such as rising lactate and six hours of elevated heart rate, rather than the model's internal reasoning.
  • Display uncertainty as confidence bands so clinicians can see when the model is guessing.
  • Design the tool so the clinician stays the decision maker and the AI behaves like a well-read intern that can be overruled.
  • Treat vendor-published accuracy numbers, including OpenAI's medical benchmarks, as marketing until an independent study checks them.
  • Before approving any clinical AI, sit a frontline clinician in front of it and ask whether they can tell when it is wrong.
Read the full transcript

Welcome to the Leaders Insights podcast.

Today's episode, why frontline clinicians don't trust clinical AI and what actually

changes that.

Here's something that should embarrass the entire clinical AI industry.

Most of these tools don't fail because the model is wrong.

They fail because nobody in the room knows when to believe it.

Right.

And that's the part everyone gets backwards.

Nurses obsess over accuracy on a validation set, a curated batch of test cases where the

model already knows the answer.

Then it hits a real ward and the nurse just… ignores it.

Why ignore it though?

If the thing is right 94% of the time, isn't skepticism the irrational move?

Because right 94% of the time is useless at the bedside.

The clinician doesn't get a lifetime average.

They get one patient, one alert, right now.

And no way to tell if this is one of the 94 or one of the 6.

So it's a trust problem, not a math problem.

It's a calibration problem.

Calibration meaning.

Does the model know when it doesn't know?

A good system says, I'm 60% confident, and here's why I'm shaky.

A bad one barks the same confident tone, whether it's certain or guessing.

Clinicians spot that fast, and once they catch it lying with a straight face once,

Give me the concrete example.

Where did this actually blow up?

The classic is Epic's sepsis prediction model.

Sepsis being that runaway infection response that kills fast.

It got deployed across hundreds of US hospitals.

An external study out of Michigan found it missed most sepsis cases and flooded staff

with false alarms.

And here's the durable lesson.

The failure wasn't just accuracy.

It was that clinicians had no window into why it fired.

So they turned it into wallpaper.

Alert fatigue.

Alert fatigue, which is a polite way of saying you trained your best people to ignore the software.

That's worse than no software, because now you've spent the budget and eroded trust for the next tool too.

Okay, so explainability is the fix.

But explainability is one of those words that means everything and nothing.

What does it actually look like on a screen?

Two things and they're humble.

One, show the evidence not the reasoning.

Don't tell me the neural network's internal weights.

Tell me flagging this because lactate is climbing and heart rate's been up for six hours.

That's a sentence a doctor can argue with.

Two, show uncertainty honestly.

Confidence bans, not a single number pretending to be gospel.

You said argue with.

That's interesting.

You want the doctor to be able to overrule it.

Every time.

The tools that get adopted treat the clinician as the decision maker

and the AI as a loud, well-read intern.

The ones that get abandoned try to be the attending physician.

Nobody trusts an intern who won't show their work.

Now, the numbers.

I know OpenAI has been pushing figures on clinical accuracy this year.

They have and their medical benchmark results are genuinely strong.

But flag it.

OpenAI is an AI lab selling the models.

So treat their internal scores as a sales brochure until someone independent checks them.

On the adoption side, I lean on O'Reilly's radar surveys, which track what practitioners

actually deploy, not what vendors wish they'd deploy.

And the gap between model performs and model gets used is enormous.

How enormous.

Ballpark.

O'Reilly's work has consistently shown that the blocker isn't model quality.

It's trust, integration, and workflow.

A brilliant model bolted onto a broken workflow dies.

A decent model that explains itself and fits how people already work gets used daily.

So the operators who win aren't the ones with the best model.

They're the ones who understood before GoLeave that trust is a feature you design, not

a thing that shows up after launch because your accuracy is high.

The best health system AI leaders run a trust review alongside the clinical validation.

Separate sign off?

Give me the one thing a listener does Monday morning.

Before you approve any clinical AI, sit a front line clinician in front of it and ask,

can you tell me when this is wrong?

If they can't answer, you don't have a tool.

You have expensive wallpaper.

Sources for today's episode.

TechCrunch AI, Ars Technica AI, The Decoder, O'Reilly Radar, OpenAI, Vendor, AI Lab,

KD Nuggets.

Done.

Want to know where you actually stand?

Take the AI self-assessment at MBA-training.com.

Read and go deeper