Building accountability and transparency into govtech AI

The framework below has three pillars: assess before you deploy, document what you deployed, and let people contest decisions.

Pillar 1: Algorithmic Impact Assessments (AIAs)

An Algorithmic Impact Assessment (AIA) is a structured review, done before deployment, that documents what a system does, who it affects, and what could go wrong. Think of it as an environmental impact statement for code.

Canada's federal government offers the best free working example. Their Directive on Automated Decision-Making requires an AIA questionnaire that scores each system from Level I (little impact) to Level IV (high impact, like decisions affecting rights or health). Higher levels trigger stricter requirements: peer review, human-in-the-loop, more explanation.

A usable AIA answers:

• What decision is automated, and is a human reviewing it? Fully automated benefit denial is riskier than a tool that flags cases for a caseworker.
• Who is affected, and are impacts distributed unevenly? Test outcomes across race, gender, age, and disability.
• What data trains and feeds the model? Historical arrest data, for example, encodes past policing patterns, not future crime.
• What is the fallback when the system fails? The A-level fiasco had none.

Do the AIA early and publicly

Run the AIA during procurement, not after launch. Write "AIA required, results published" into the request for proposals so vendors price it in. Publish a plain-language summary. If you cannot explain a system to a city council, you should not deploy it against residents.

Pillar 2: Public model registries

A model registry is a public inventory of the automated systems an agency uses. It turns "black box government" into something a journalist, auditor, or resident can inspect.

Amsterdam and Helsinki pioneered public AI registers listing each algorithm, its purpose, the data it uses, and a contact for questions. New York City passed Local Law 49 of 2018 creating an automated decision systems task force, and several US cities now maintain public inventories.

A strong registry entry includes:

| Field | Example |

|---|---|

| System name and owner | Benefits Eligibility Screener, Dept. of Human Services |

| Purpose | Prioritize applications for caseworker review |

| Decision type | Advisory (human decides) |

| Data sources | Application form, income records |

| Risk level | AIA Level III |

| Vendor and model | Vendor X, logistic regression |

| Last audit date | 2025-11 |

| Appeal contact | ombudsman@agency.gov |

Keep it machine-readable. A public registry is not just a PDF; it should be queryable so oversight bodies can spot patterns across agencies.

This snippet is a single registry record. Publishing records in a standard format lets a state comptroller run one query across every department instead of filing 30 records requests.

Pillar 3: Appeal and due-process mechanisms

Assessment and documentation are useless if a citizen cannot fight a wrong decision. This is where most govtech AI still fails.

Due process in an automated context requires four things:

1. Notice. Tell the person an automated system was involved. "Your application was scored by an eligibility model" is the minimum.

2. Explanation. Give the specific reasons, not "the model said no." A denied applicant should learn which factors drove the outcome.

3. Human review on appeal. A person, not the same algorithm, reviews contested cases. The EU's GDPR (Article 22) already grants a right to human review of solely automated decisions with legal effects.

4. Correction and feedback. If an appeal reveals bad data, fix the record and check whether others were affected the same way.

A concrete appeal flow

Imagine an unemployment agency using a model to flag possible fraud:

• The claimant receives a notice: flagged for review, benefits paused, here are the three factors that triggered it.
• The notice includes a deadline, a form, and a human contact.
• A caseworker (not the model) reviews evidence within a set service standard, say 10 business days.
• If the flag was wrong, benefits are restored with back pay, and the agency logs the error to retrain the model.

Michigan's MiDAS unemployment system is the cautionary tale here: it wrongly accused tens of thousands of fraud with almost no human review, and the state paid large settlements. The failure was not the math. It was the absence of an appeal path.

Tying it together: governance as a lifecycle

These three pillars are not a one-time checklist. They form a loop.

Before deployment: Run the AIA. Set the risk level. Decide whether a human must stay in the loop.

At deployment: Publish the registry entry. Announce the appeal channel.

During operation: Monitor for drift (when a model's accuracy degrades as real-world conditions change) and disparate impact. Publish audit dates.

After incidents: Feed appeal outcomes back into retraining. Update the registry. If harm is severe, pause the system, as the UK did with A-levels, but ideally before launch.

Who owns each pillar

Accountability collapses when "everyone" is responsible. Assign named roles:

• A senior accountable official who signs off on the AIA and can be summoned by the legislature.
• A data or algorithm review board with technical and community members.
• An ombudsman or appeals unit independent of the team that built the system.

Independence matters. The people who deploy a model should not be the sole judges of appeals against it.

Handling the vendor problem

Most agencies buy AI rather than build it. Vendors often claim their models are proprietary. Defeat this in the contract, not in court:

• Require the vendor to complete the AIA and permit independent audits.
• Require that documentation sufficient for FOIA disclosure be delivered and owned by the agency.
• Reject "the model is a trade secret, so residents cannot see how they were scored." A public decision must be publicly explainable.

The US OMB memo M-24-10 on federal AI use codifies much of this for federal agencies, including required impact practices and public inventories. Use it as a template even if you are a nonprofit or a city.

Key Takeaways

• Assess before you deploy. An Algorithmic Impact Assessment run during procurement, with a published plain-language summary, catches the failures that scrapped the UK A-level algorithm after the fact.
• Inventory publicly. A machine-readable model registry (see Amsterdam and Helsinki) turns records requests into queries and makes hidden systems visible to auditors and journalists.
• Guarantee appeal. Notice, specific explanation, independent human review, and correction are the minimum due-process floor. The absence of appeal, not bad math, caused Michigan's MiDAS scandal.
• Assign named owners. A senior accountable official, a review board, and an independent ombudsman prevent the "everyone and no one is responsible" trap.
• Beat vendor secrecy in the contract. Require AIAs, audit rights, and FOIA-ready documentation as procurement terms. A public decision must be publicly explainable.