Auditing algorithmic bias in benefit and enforcement systems

The core test: measuring disparate impact

Start with a simple, defensible metric. Two are standard.

Selection rate. For each group, what share got the adverse outcome (flagged, denied, audited)? Compare rates across groups.

The classic benchmark is the four-fifths rule from US employment law: if a protected group's selection rate is less than 80 percent of the most-favored group's rate, that is evidence of adverse impact worth investigating. It is a screening tool, not a legal verdict, but it is a useful trigger.

Here is a minimal audit in Python using a fairness library:

If Group A is flagged at 6 percent and Group B at 20 percent, the ratio is 0.30. That is far below 0.80 and demands explanation.

Selection rate is not enough

A lower flag rate is not automatically fair, and equal flag rates are not automatically fair either. You also need to check error rates, because the real harm is being *wrongly* flagged.

• False positive rate: of people who did nothing wrong, what share got flagged? If innocent members of one group are flagged twice as often, that is a serious equity problem even if overall flag rates look balanced.
• False negative rate: of actual fraud cases, what share were missed?

In the Dutch case, the deeper scandal was the false positives: honest families treated as fraudsters. Always audit both.

Fairlearn is a free, open-source toolkit that computes these metrics and is a reasonable starting point for a non-vendor audit.

Proxies: the trap that catches everyone

Suppose you remove nationality and race from your model to be safe. You are not safe.

Models reconstruct protected attributes from proxy variables: features that correlate with group membership. Postal code correlates with race and income. Language of correspondence correlates with national origin. Name spelling, browser language, even "number of prior contacts with the agency" can all encode the very trait you tried to exclude.

The Dutch model reportedly treated dual nationality and low income as risk signals. Even where an attribute was later removed, correlated features can carry the same discriminatory signal forward.

Practical step: test whether you can predict the protected attribute from the remaining features. If a simple model predicts nationality from your "neutral" inputs with high accuracy, those inputs are proxies. Investigate them.

Building the audit into procurement

Most agencies do not build these models. They buy them. That means your strongest control point is the procurement contract: the terms under which you purchase the system.

Vendors will resist disclosure, citing trade secrets. Push back in writing before signing. Concrete clauses to require:

• Bias testing disclosure. The vendor must deliver selection-rate and error-rate breakdowns across protected groups on your population, not a generic benchmark.
• Training data description. Where the data came from, what period it covers, and known gaps or skews.
• Right to audit. You (or an independent third party) can test the model on held-out data, including access to inputs and outputs.
• Explainability requirement. For any adverse decision, the system must produce a human-readable reason.
• Human-in-the-loop guarantee. No fully automated adverse decision without a trained caseworker able to override it.
• Update notification. The vendor must tell you when the model changes, because a retrained model can introduce new bias.

The EU AI Act classifies systems that determine access to public benefits and services as high-risk, triggering obligations around risk management, data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.Voir la définition complète →, transparency, and human oversight. If you operate in or sell to the EU, these are not optional in 2026. Even outside the EU, they are a sound baseline.

🎬 [VIDEO: "The Dutch benefits scandal explained" — youtube.com — a concise overview of how the childcare benefits algorithm caused mass harm]

Governance: who is accountable

An audit is only as good as the authority behind it. Set these up before deployment.

• A named human owner for the model, accountable for its decisions. Not "the algorithm decided."
• An appeals path that is fast and does not require a lawyer. In the Dutch case, families fought for years to clear their names.
• Logging. Record every decision, the inputs, and the score, so you can reconstruct what happened when someone contests it.
• Regular re-audits. Bias drifts as populations and data change. Audit on a schedule, not once.

A short worked scenario

Your agency deploys a model to prioritize which benefit claims get a manual fraud review. After three months you pull the logs.

You find:

• Applicants from immigrant-heavy districts are flagged at 18 percent; other districts at 7 percent. Ratio: 0.39.
• The false positive rate for the higher-flagged group is 22 percent versus 9 percent for others.

What do you do?

1. Do not shut off review entirely. That could let real fraud through and is its own failure.

2. Trace the driver. Which features push the score up for flagged groups? Often it is a proxy like postal code or "incomplete documentation," which correlates with language barriers, not fraud.

3. Test a version without the suspect features and compare fraud-detection performance. If you lose little detection but cut the disparity sharply, the feature was doing bias work, not fraud work.

4. Add a human review floor. No clawback without caseworker sign-off and a written reason the applicant can see.

5. Document everything for your regulator and your appeals process.

This is the difference between a defensible system and a scandal: you measured, you traced, you fixed, and you can show your work.

Key takeaways

• Measure outcomes, not intentions. Compute selection rates and, critically, false positive rates across protected groups. The four-fifths (0.80) ratio is a screening trigger, not a verdict.
• Removing protected attributes does not remove bias. Proxy variables like postal code and language reconstruct it. Test whether your "neutral" features predict the protected trait.
• Procurement is your leverage. Require bias disclosures, audit rights, explainability, and human override in the contract, before you sign.
• Keep a human accountable and a fast appeals path open. The Dutch scandal's cruelty was the years families spent unable to contest a machine's verdict.
• Audit on a schedule. Retrained models drift. A system that was fair last year may not be fair today.