# Setting realistic pilot timelines and success metrics
A state unemployment office deploys an AI document-processing tool to read scanned unemployment claims, extract wage data, and flag inconsistencies for human review. Month one: promising demo results. Month six: still "piloting." Month eighteen: nobody can say whether claims are processed faster, whether error rates dropped, or whether the tool paid for itself. The pilot became permanent, not because it succeeded, but because nobody defined what success meant before it started.
This is the single most common failure mode in public sector AI adoption: pilots without exit criteria. This lesson gives you a concrete 6-month framework to avoid it.
Three structural reasons public sector AI pilots run indefinitely:
1. No pre-agreed go/no-go metrics. Success is defined retroactively, so any result can be spun as "promising."
2. Procurement sunk cost. Once a vendor contract is signed (often through a lengthy RFP, or Request for Proposal, process), agencies feel pressure to justify the spend rather than kill the project.
3. Diffuse accountability. In agencies, no single owner is penalized for an indefinite pilot, unlike in a private firm where a P&L (profit and loss) owner faces pressure to show returns.
The fix is not more caution. It's more structure, defined upfront.
Before touching the AI tool, measure the status quo. You cannot claim improvement without a baseline.
For the unemployment claims example, capture:
This is also when you define the go/no-go metrics in writing, signed off by the program owner and IT/data leadership before any AI output is reviewed. Metrics agreed to after seeing results are not metrics, they're rationalizations.
Run the AI tool on a defined subset: a specific claim type, a specific regional office, or a fixed volume (e.g., 2,000 claims). Keep a parallel control group processed the old way if staffing allows. This is the closest a public agency gets to an A/B testA/B testA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.View full definition →.
Track weekly, not just at the end:
Compare Phase 2 results against the Phase 1 baseline and the pre-agreed thresholds. Three outcomes only:
The critical discipline: "extend with cause" requires naming the exact problem and the exact new checkpoint date. "Let's give it more time" is not a valid Phase 3 outcome.
Vague metrics like "improves efficiency" are not testable. Use this structure instead.
| Metric | Baseline (example) | Go threshold (example) | How measured |
|---|---|---|---|
| Processing time/claim | 22 minutes | ≤ 14 minutes | System timestamps |
| Field extraction accuracy | n/a (manual) | ≥ 95% | Sample audit vs. human review |
| Human override rate | n/a | ≤ 20% of claims | Caseworker logs |
| Cost per claim | Agency-specific, calculate locally | 15% reduction | Staff hours x loaded rate |
Worked example: If a caseworker's loaded hourly cost (salary plus benefits and overhead) is an estimated $45/hour (illustrative, verify locally), and manual processing takes 22 minutes (0.367 hours), cost per claim is:
0.367 hours x $45/hour = $16.50 per claimIf the AI tool cuts processing to 14 minutes (0.233 hours):
0.233 hours x $45/hour = $10.50 per claimThat's a $6 per claim reduction. At 50,000 claims a year, that's an estimated $300,000 in annual labor cost avoidance, before subtracting the tool's licensing and maintenance costs. This is the kind of simple, defensible calculation a program owner needs to bring to budget committees or legislative oversight hearings.
Even a technically successful pilot can fail on grounds that matter more in government than in the private sector:
Build a simple check for each guardrail into the Phase 3 decision, not as an afterthought.
Knowledge check
1. In the unemployment office example, why did the AI pilot become permanent rather than reaching a clear conclusion?
2. Why is establishing a baseline during Phase 1 (Weeks 1-4) essential before evaluating an AI pilot's impact?
3. A public agency has signed a costly RFP-based vendor contract for an AI pilot. Six months in, results are mediocre, but leadership is reluctant to end the pilot. Which structural driver of pilot drift does this best illustrate?
4. Select ALL correct answers about why public sector AI pilots tend to run indefinitely.
Select all the correct answers.
5. Select ALL correct answers about what should be measured or established as part of the baseline in Phase 1 of the 6-month framework.
Select all the correct answers.
A no-go decision is not a failure of the pilot process, it's the process working. Document three things when killing a pilot:
1. Which specific metric(s) were missed, and by how much
2. Whether the shortfall was a tool problem (accuracy too low) or an integration problem (data formats incompatible with legacy systems)
3. Whether a different, narrower use case within the same tool could still be viable (e.g., the tool fails on handwritten claims but works well on employer-submitted digital wage reports)
This documentation matters because it turns a "failed" pilot into reusable institutional knowledge for the next procurement cycle, rather than a buried embarrassment nobody discusses.
🎬 [VIDEO: "How Government Agencies Are Piloting AI" - youtube.com - search for GAO (Government Accountability Office) or state CIO panel discussions on AI pilot governance, which walk through real public sector pilot structures and oversight questions]
Vendors selling document AI often cite accuracy figures from their best-case client deployments, frequently in the 90-98% range for structured form extraction (estimates, vary widely by document quality and vendor). Public sector documents are messier: handwritten forms, multiple languages, scanned faxes. Expect real-world accuracy to run lower than vendor marketing materials during early pilot phases. Build that gap into your go/no-go thresholds rather than being surprised by it in Month 4.