The algorithm that denied bail: what a 2016 courtroom controversy still teaches us about AI fairness

August 23, 2026 · 5 min · Free, with full transcript

In 2016, an algorithm called COMPAS was put under the microscope by ProPublica journalists, and what they found split the AI community down the middle. The argument that followed is one of the clearest illustrations of why "fairness" in AI is not a technical setting you dial in, but a choice with real consequences.

Listen to the podcast

4 min

Read the full transcript

Welcome to the Leaders Insights Podcast. Today's episode, the algorithm that denied bail what

a 2016 courtroom controversy still teaches us about AI fairness. A decade ago, a piece

of software helped decide whether people walked free or sat in a jail cell. And we still can't

agree on whether it was fair. Why does that old fight matter in 2026?

Because we never actually settled it. The tool was called COMPASS, Correctional Offender

Management Profiling for Alternative Sanctions, a mouthful that just means a risk score judges

used to guess if a defendant would re-offend. Higher score, less likely you get bail. And

in 2016, ProPublica, the investigative outlet, tore into it. What did they find? They found

that black defendants, who did not go on to re-offend, were roughly twice as likely to

be wrongly flagged high risk than white defendants in the same boat. 45% versus 23% in their

numbers. That sounds like an open and shut case of a racist algorithm. Done. Next topic.

That's what everyone thought. Then Northpoint, the company behind it, fired back with their

own numbers. And here's the twist. They were also right. They can't both be right.

That's not how being right works. Welcome to fairness in AI. Northpoint said,

Look, when we call someone high risk, we're correct about the same percentage of the

time, regardless of race. A high score meant roughly the same real-world probability for

a black defendant as a white one. That's called calibration. The score means what it says,

consistently. So the score is honest, but the mistakes land unevenly. How does that

happen at the same time? Because when the base rates differ, when arrest rates in the

underlying data differ between groups, you can hold one kind of fairness or the other,

but not both. It's a mathematical trade-off. A researcher named Kleinberg proved it. You

literally cannot satisfy both definitions unless the groups are identical to begin with.

Give me that in plain English. Because that sounds like a cop-out. Picture two buckets

of apples. One bucket has more bruised apples than the other. If you calibrate your sorting

machine so labeled bad means the same bruise probability in both buckets, you'll end up

wrongly tossing more good apples from the bruised bucket. Fix that error gap, and now,

labeled bad means different things depending on the bucket. Pick your poison. So the engineers

didn't screw up. The universe did. The engineers screwed up by pretending it was a purely technical

question. There is no default setting labeled fair. Someone has to choose which error matters

more. And that's a moral call dressed up as a math problem. Okay, but who should make

that call? Because right now it sounds like it fell to a private company by accident. Exactly

the scandal. Northpoint decided the trade-off in a boardroom, and it took journalists to

surface it years later. The lesson for any executive deploying AI in 2026. The moment

your model touches people's lives, loans, hiring, insurance, you are making a fairness

choice whether you name it or not. Silence is a choice too, just a cowardly one. Ten

years on, has anyone actually fixed this? The EU AI Act is live now. We've got all

this governance machinery. The machinery helps. The EU AI Act, in force this year, forces

high-risk systems to document exactly these trade-offs. But documentation isn't a decision.

I've watched compliance teams write beautiful 10-page fairness reports that never once

say, we chose to protect against this error instead of that one. They describe the fork

in the road and refuse to say which way they turned. So the paperwork exists to prove you

thought about it, not to prove you chose well. Right, and regulators can't pick the

trade-off for you because it depends on context. In criminal justice, a false high-risk flag

steals someone's freedom. That error is monstrous. In cancer screening, a false all-clear

kills someone. Different stakes, different choice. Give me the one thing an exec should

do Monday morning. Before you ship any model that judges people, write one sentence. When

we're wrong, we'd rather be wrong in this direction because if your team can't finish

that sentence, you're not ready to deploy. You're just automating a decision nobody's

willing to own. That's it. The AI Decision Tools are live

at MBA-training.com.

Read and go deeper