The algorithm that denied bail: what a 2016 courtroom controversy still teaches us about AI fairness

The defendant's name was Dylan Fugett. He had a minor drug charge on his record. The algorithm assessing his recidivism risk rated him low. He went on to commit further offences. Across the same Florida county, a Black woman named Brisha Borden, who had no prior adult convictions, was rated high risk after being caught riding a stolen bicycle. She did not reoffend.

These two cases, surfaced by ProPublica in a May 2016 investigation, became the most debated AI fairness case on record. The system at the center of it was COMPAS, built by a company called Northpointe (later rebranded Equivant). Courts across the United States were using its scores to inform decisions about bail, sentencing, and parole.

What actually happened

ProPublica's journalists analyzed roughly 7,000 risk scores assigned by COMPAS in Broward County, Florida, and cross-referenced them with actual recidivism data two years out. Their finding was stark: Black defendants were nearly twice as likely as white defendants to be falsely labeled high risk, while white defendants were more likely to be incorrectly flagged as low risk when they went on to reoffend.

Northpointe pushed back, and with some technical justification. The company argued that COMPAS was "fair" by a different definition: within each racial group, the score's predictive accuracy was roughly equal. If a Black defendant and a white defendant both received a score of seven out of ten, they had similar actual recidivism rates. By that measure, the algorithm was calibrated correctly.

Here is the uncomfortable part: both sides were right on their own terms. A group of academic researchers, including Jon Kleinberg at Cornell, Sendhil Mullainathan at the University of Chicago, and Christoph Lassner, published a paper showing mathematically that when the base rates of an outcome differ across groups, you cannot simultaneously satisfy all common definitions of fairness. You must choose. Equalizing false positive rates, equalizing false negative rates, and maintaining calibration across groups are, in most real-world conditions, mutually exclusive. This result was not hypothetical. It was a theorem.

The COMPAS debate made that theorem visible to anyone watching, including journalists, judges, and eventually regulators.

Why it still matters

The COMPAS case exposed something that the industry had been slow to admit publicly: there is no view from nowhere in AI fairness. Every fairness criterion embeds a value judgment about which errors are most costly and to whom. A system optimized to minimize false positives for one group will, almost inevitably, increase false negatives for another. Choosing between them is a policy decision, not an engineering problem.

This matters well beyond criminal justice. Credit scoring systems, hiring tools, healthcare triage algorithms, and insurance pricing models all face versions of the same dilemma. In 2018, Amazon scrapped an internal recruiting tool after discovering it systematically downgraded CVs from women, apparently because it had been trained on a decade of historical hiring data that skewed male. The training data did not contain explicit gender labels, but the model found proxies: the names of women's colleges, certain verbs common in female-coded writing styles. Removing the feature did not fix the problem because the signal was distributed throughout the data.

In healthcare, a widely cited 2019 study published in Science by Ziad Obermeyer and colleagues found that a commercial algorithm used by US health systems to allocate care management resources was significantly less likely to flag Black patients as high-need compared to equally sick white patients. The reason was indirect: the algorithm used healthcare costs as a proxy for health needs, and because systemic inequities meant Black patients had historically spent less on care, the model interpreted lower spending as lower need. The company involved, Optum (a UnitedHealth Group subsidiary, so a vendor source), disputed some characterizations but worked with researchers to adjust the model.

None of these systems were designed to discriminate. That is the point.

The takeaway for you

If you are a professional deploying or commissioning an AI system that affects people, the question "is this model accurate?" is necessary but not complete. You need to ask a second question: accurate for whom, and at whose expense when it is wrong?

This requires three things that are often skipped under delivery pressure.

First, disaggregate your validation metrics. An overall accuracy of 87% can hide a model that performs well for the majority group and badly for everyone else. Break down error rates by every demographic dimension that matters in your context. If your vendor cannot provide this breakdown, that is itself a signal.

Second, be explicit about which fairness criterion you are applying and why. Document the choice. "We prioritized equalizing false positive rates across groups because the cost of wrongly denying access falls disproportionately on applicants" is a defensible position. "We didn't think about it" is not.

Third, track outcomes after deployment. Training data reflects the past. If your model shapes decisions that feed back into future data, bias can compound quietly over time. This is not a theoretical concern: it has been observed in predictive policing systems, where areas flagged as high-risk received more patrols, which generated more arrests, which reinforced the model's predictions.

The COMPAS story did not end with a resolution. Courts still use risk assessment tools. The fairness debates continue. What changed is that "the algorithm said so" became a much harder justification to hide behind. Professionals who understand why fairness criteria conflict with each other will be better positioned to ask the right questions before a system goes live, rather than defending it after it makes the front page.