The pre-deployment checklist before AI touches money

Step 1: Red-teaming the model before it sees a real applicant

Red-teaming means deliberately attacking your own model to find where it breaks, discriminates, or can be gamed.

For a lending model, a red team asks:

• Can we reconstruct a protected characteristic (race, gender, disability status) from allowed variables like zip code, shopping patterns, or device type? This is the proxy discrimination problem, well documented in mortgage lending research from the Federal Reserve.
• What happens to approval rates when we swap only the applicant's name or zip code, holding income and credit history constant? This is a standard disparate impact test.
• Can adversarial applicants manipulate inputs (padding income data, timing applications) to flip a decision?

A concrete check: run the model against a held-out test set segmented by race, gender, and age proxy groups, and compare approval rates. US regulators use the four-fifths rule (a group's selection rate should be at least 80% of the highest group's rate) as a rough screening threshold, originally from EEOC employment guidance but widely borrowed in fair-lending analysis.

Step 2: Human-in-the-loop thresholds

Human-in-the-loop (HITL) means a person reviews or can override the model before the decision executes. The governance question is never "should humans be involved," it's "at what threshold."

A typical committee sets tiered thresholds:

| Decision confidence | Action |

|---|---|

| High-confidence approval (model score above 0.9) | Auto-approve, logged for periodic audit |

| Mid-range score (0.4 to 0.9) | Routed to human underwriter |

| High-confidence denial | Auto-deny, but with mandatory adverse action notice and human spot-check sampling |

| Edge cases (thin credit file, first-time borrower) | Always human-reviewed |

Under the US Equal Credit Opportunity Act, denied applicants must receive an adverse action notice stating specific reasons. If a model denies credit, someone has to translate the model's internal logic into a human-readable reason. This is why many lenders pair complex models with explainability tools like SHAP (SHapley Additive exPlanations), which attribute a prediction to individual input features.

If the top reasons include something like "device operating system" or an opaque embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition → dimension, that's a governance red flag, not a compliant adverse action reason.

Step 3: Rollback plans

A rollback plan answers: if this model misbehaves in production, how fast can we revert to the previous known-good version, and what happens to decisions made in between?

Committees demand answers to three things before sign-off:

1. Kill switch latency: how many minutes between detecting an anomaly and disabling the model. Leading fintechs target under 15 minutes for automated shutdown triggers.

2. Champion-challenger setup: run the new model in shadow mode (scoring real applications without acting on them) against the current production model for a defined window, often 30 to 90 days, comparing outcomes before full cutover.

3. Remediation for affected decisions: if 4,200 applicants were wrongly rejected, is there a process to re-score them and issue corrected decisions? Regulators increasingly expect this, and the EU AI Act requires "human oversight" measures precisely so this remediation path exists (Article 14).

Step 4: Audit trails

An audit trail is the immutable record of what the model did, on what data, under what version, and who approved it.

Minimum viable audit trail for a lending model:

• Model version ID and training data snapshot date
• Every input feature used for a specific decision
• The output score and the final decision (including any human override, with the overriding officer's ID)
• Timestamp and model card reference (a model card is a standardized document, a concept popularized by Google researchers in 2019, summarizing intended use, performance across subgroups, and known limitations)

This is not optional paperwork. The CFPB has already brought enforcement actions against lenders for "black box" models that couldn't produce specific denial reasons, and the EU AI Act mandates logging capabilities for high-risk systems (Article 12) with records retained for a minimum period specified in implementing guidance.

Step 5: The sign-off gate

By the time a model reaches committee, it should arrive with a governance package, not a pitch deck. A realistic sign-off checklist:

• [ ] Fair-lending disparate impact test completed, results within threshold (e.g., four-fifths rule) or flagged with mitigation plan
• [ ] HITL thresholds defined and coded into the decision pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition →, not left to informal practice
• [ ] Shadow-mode results reviewed for at least one full business cycle
• [ ] Rollback and kill switch tested, not just documented
• [ ] Audit logging verified end-to-end with a sample reconstruction of five real decisions
• [ ] Model card published internally and shared with compliance and legal
• [ ] Named accountable owner (a real person, not "the ML team") who signs off

This last point matters more than it sounds. Both the EU AI Act and US regulatory guidance (see the Federal Reserve's SR 11-7 model risk management guidance, still the reference framework used across US banking supervision) push toward named accountability rather than diffuse team ownership. When something breaks, "the model did it" is not an answer regulators accept.

🎬 [VIDEO: "Explainable AI in Financial Services" - youtube.com - a walkthrough of how banks use SHAP and LIME to make credit models explainable to regulators and customers]

Key Takeaways

• Red-teaming before deployment means actively testing for proxy discrimination and disparate impact, not just accuracy metrics; use quantitative screens like the four-fifths rule as a starting point, not a finish line.
• Human-in-the-loop is a tiered design decision (confidence thresholds determine when a human reviews or overrides), not a binary "humans are involved somewhere" claim.
• Rollback plans need a tested kill switch, a shadow-mode comparison period, and a remediation path for decisions made while the model was misbehaving.
• Audit trails must let you reconstruct any single decision (inputs, version, output, human override) years later, since regulators like the CFPB and frameworks like the EU AI Act require exactly this traceability.
• A governance sign-off is only real if there's a named, accountable human owner and a documented package, not an informal "the model looked fine in testing" conversation.