# The pre-deployment checklist: red-teaming government AI
A city planning office in Ohio quietly rolled out an AI tool to pre-screen home renovation permits in 2023. Within weeks, residents in one zip code were getting automatic rejections at nearly triple the rate of a wealthier neighborhood across town, for the same type of request. Nobody had tested that scenario before launch. This lesson walks through the checklist that should have existed, using a permitting AI as the running case study.
Permitting decisions (building permits, business licenses, zoning variances) sit at a sensitive intersection: they are high-volume, rules-based enough to tempt automation, but consequential enough that a bad call blocks someone's home renovation or livelihood.
This makes permitting AI a useful lens for the whole toolkit of pre-deployment governance. The same checklist applies to benefits eligibility, fraud detection, or risk scoring in criminal justice.
Government AI deployments in the US now typically fall under agency-specific AI governance frameworks following the White House Office of Management and Budget (OMB) Memorandum M-24-10 (2024), which requires federal agencies to designate Chief AI Officers and conduct risk assessments for "rights-impacting" AI, a category that explicitly includes permitting and licensing decisions. In the EU, the EU AI Act (entered into force 2024, phased obligations through 2027) classifies systems that determine access to public services as "high-risk," triggering mandatory conformity assessments before deployment.
Before any model touches data, write down precisely what the system does and does not decide.
Concrete rule of thumb: if a human's home, income, or liberty is affected, the AI should recommend, not decide, unless a statute explicitly authorizes automated final action. This is the "human-in-the-loop" versus "human-on-the-loop" distinction, and it is not just semantics; it changes your legal exposure entirely.
Every training dataset for a permitting model has history baked in. If historical permit approvals reflect decades of uneven code enforcement across neighborhoods, the model learns that pattern as "normal."
Checklist items:
A simple disparate impact check, often called the "four-fifths rule" (a US Equal Employment Opportunity Commission guideline adapted widely in algorithmic auditing), flags a problem if one group's selection rate is less than 80% of the highest group's rate.
# Simple four-fifths rule check
approval_rate_group_a = 0.72
approval_rate_group_b = 0.51
ratio = approval_rate_group_b / approval_rate_group_a
print(f"Impact ratio: {ratio:.2f}")
if ratio < 0.8:
print("FLAG: potential disparate impact, investigate before deployment")Here, 0.51 / 0.72 = 0.71, below the 0.8 threshold. That is a stop-ship signal, not a footnote for the appendix.
"Red-teaming" means deliberately attacking your own system to find failure modes before an adversary, journalist, or lawsuit does. Borrowed from military and cybersecurity practice, it is now a named requirement in NIST's AI Risk Management Framework (AI RMF, 2023), the closest thing the US has to a technical standard for trustworthy AI.
For permitting AI, red-teaming means actively trying to:
Practical technique: build a red-team test set deliberately weighted toward the messy 10% of real applications, not the clean 90% used for training accuracy metrics. A model boasting 95% accuracy overall can still fail 60% of the edge cases that matter most.
Run the model in "shadow mode": let it score real incoming applications, but keep human decision-makers fully in charge, and compare outcomes for weeks or months before switching it on for real.
This surfaces problems no lab test catches: seasonal spikes (spring renovation season), local ordinance changes the model wasn't retrained on, or drift as application language evolves.
Document a pre-registered set of pass/fail thresholds before the shadow period starts (for example: disparity ratio must stay above 0.8, false-denial rate on appealed cases must stay under a set ceiling). Deciding thresholds after seeing results invites motivated reasoning.
Vérification des acquis
1. Why is permitting AI used as a representative case study for government AI governance more broadly, rather than being treated as a narrow niche issue?
2. What was the core governance failure illustrated by the Ohio permitting AI example, where one zip code saw automatic rejections at a much higher rate than another?
3. Why does 'Gate 1: Define the decision the AI is actually allowed to make' matter as a first step in pre-deployment governance?
4. Select ALL correct answers about why permitting decisions are described as sitting at a 'sensitive intersection' in government AI deployment.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about the regulatory context described for government AI in permitting and licensing.
Sélectionnez toutes les réponses correctes.
Every rights-impacting government AI system needs a designed, tested override path, not a hidden "email the office" workaround.
What good override design looks like:
The EU AI Act codifies this explicitly, requiring "human oversight" measures for high-risk systems, including the ability for a human to override or disregard the system's output entirely (Article 14).
Pre-deployment testing is necessary but not sufficient. Set up:
Agencies that skip this step often discover bias only after a local news investigation or a lawsuit, which is a far more expensive way to find out.
🎬 [VIDEO: "The UK's A-Level Grading Algorithm Scandal" - youtube.com - a widely cited case study on what happens when an automated scoring system replaces human judgment without adequate red-teaming, search for coverage from BBC or Guardian explainers]