# Evaluating AI vendors against public sector procurement criteria
A regional housing authority receives an RFP (Request for Proposal) response from an AI vendor promising to automate eligibility screening and case triage for 40,000 tenant files. The deck says "99% accuracy" and "enterprise-grade security." The procurement officer has thirty minutes to decide whether this claim survives contact with a security audit. This lesson is that thirty minutes.
Private companies can pilot an AI tool on a handshake and a purchase order. Public agencies cannot. Most US public sector AI purchases must pass through formal procurement rules: state IT security standards, federal FedRAMP requirements for cloud services touching federal data, and increasingly, state-level AI-specific procurement policies (California, Texas, and Connecticut have all issued state AI procurement guidance as of 2025).
FedRAMP (Federal Risk and Authorization Management Program) is the US government's standardized process for assessing and authorizing cloud products for federal use. If a vendor's product isn't FedRAMP-authorized and the use case touches federal data (e.g., HUD-funded housing programs), that's a disqualifying gap, not a negotiable detail. Check the FedRAMP Marketplace directly rather than trusting a vendor's slide.
State and local agencies often use parallel frameworks like StateRAMP, built on similar principles for state and municipal government cloud purchases.
The point: procurement criteria aren't bureaucratic friction. They're the mechanism by which a public agency avoids putting citizen data (immigration status, income, disability records) into a system with no accountability trail.
Here's a condensed version of a realistic vendor claim, followed by what a sharp evaluator asks.
> "Our AI platform achieves 99% accuracy in eligibility determination and uses enterprise-grade encryption. We are SOC 2 compliant and trusted by leading organizations."
Four claims, four follow-up questions:
1. "99% accuracy", accuracy on what, measured how?
Accuracy without a defined task and dataset is meaningless. Ask: accuracy compared to what ground truth? On which population? A model that's 99% accurate on a training set curated by the vendor tells you nothing about performance on your housing authority's actual applicant mix, which may include non-English speakers, disability accommodations, or unusual household compositions. Demand the confusion matrix (a table showing correct vs. incorrect predictions by category), not just a single number.
2. "Enterprise-grade encryption", is this a real standard or marketing language?
"Enterprise-grade" is not a certification. Ask specifically: is data encrypted at rest and in transit using AES-256 (a specific, named encryption standard)? Is encryption key management handled by the agency or the vendor? Vague adjectives should always be translated into a named, checkable standard.
3. "SOC 2 compliant", compliant, or merely audited?
SOC 2 (System and Organization Controls 2, an audit framework maintained by the American Institute of CPAs) has two report types. Type I checks whether controls exist at a point in time. Type II checks whether controls actually operated effectively over a period (usually 6 to 12 months). Type I is a much weaker signal. Ask which type, ask for the report, and ask when it expires.
4. "Trusted by leading organizations", is this relevant social proof?
A vendor trusted by retail chains for inventory forecasting tells you nothing about their fitness for handling protected tenant data under HUD (Department of Housing and Urban Development) program rules. Ask for public sector references specifically, ideally other housing authorities or HUD grantees, and call them.
Score each vendor response across four dimensions, not just price and features.
| Criterion | What to check | Disqualifying red flag |
|---|---|---|
| Security certification | FedRAMP/StateRAMP authorization status, SOC 2 Type II report | Only "SOC 2 in progress" or no report available |
| Data residency and ownership | Where data is stored, who can access it, contract terms on data deletion | Vendor retains rights to use agency data to train models for other clients |
| Accuracy and bias evidence | Independent testing, breakdown by demographic subgroup | No third-party validation, only vendor-reported numbers |
| Explainability | Can staff see why the model flagged a case for denial or review | "Black box" model with no case-level rationale, especially for adverse decisions |
The last row matters especially in housing. Under due process principles applied to benefits determinations, an applicant denied housing assistance generally has a right to understand why. A model that can't produce a human-readable reason for a denial creates legal exposure regardless of how accurate it is on average. This is the same logic behind the NIST AI Risk Management Framework, which treats explainability and accountability as core trust requirements, not nice-to-haves.
You don't need to be a data scientist to ask for this. Request that the vendor report accuracy broken out by subgroup, not just overall. A simple way to sanity-check disparate impact:
Overall accuracy: 91%
Accuracy for English-speaking applicants: 94%
Accuracy for Spanish-speaking applicants: 78%A 16-point gap like this is a red flag regardless of the headline number. It suggests the training data underrepresented one group, which in a housing eligibility context could trigger fair housing scrutiny under the Fair Housing Act, enforced by HUD. Any evaluator, technical or not, can insist on seeing this table before scoring the "accuracy" section of an RFP response.
Vérification des acquis
1. A vendor's AI eligibility screening tool will process data from a HUD-funded housing program but is not listed on the FedRAMP Marketplace. What is the correct procurement conclusion?
2. Why can't a public housing authority simply pilot a promising AI vendor's tool the way a private company might, on a handshake agreement?
3. A vendor claims '99% accuracy' in eligibility determination for a housing authority's tenant screening. What should a sharp procurement evaluator do with this claim?
4. Select ALL correct answers about frameworks and standards relevant to public sector AI procurement.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about why a procurement officer should verify vendor claims independently rather than relying on the vendor's own materials.
Sélectionnez toutes les réponses correctes.
Vendors will pitch ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète → in terms of staff hours saved or cases processed per week. Treat these numbers as hypotheses, not facts, until validated in a pilot.
A more realistic ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète → framework for a public agency:
🎬 [VIDEO: "How Government Buys Software (and Why It's So Hard)" - youtube.com - a walkthrough of public sector procurement cycles and why compliance requirements shape vendor selection more than features do]