Running the pre-deployment gauntlet: checks that catch problems early

Concretely: a bank replacing its credit card fraud model runs the new version on every transaction for 4 to 12 weeks, logging what it *would have* flagged. Analysts compare false positive rates (good transactions wrongly blocked) and false negative rates (fraud missed) against the champion.

A simple worked example: if the champion model on 1,000,000 monthly transactions produces 2,000 false positives (blocking legitimate purchases, a 0.2% rate) and the challenger produces 2,600 (0.26%), that's a 30% increase in customer friction, even if the challenger catches more fraud. The bank must decide if the fraud-catch improvement justifies that cost. This is the kind of tradeoff a champion-challenger test surfaces before anyone experiences it live.

Step 2: Stress and scenario testing

Standard validation checks accuracy on historical data. Stress testing asks: what happens when the world looks nothing like the training data?

Banks build synthetic scenarios: a sudden rate shock, a regional recession, a novel fraud pattern, a data feed going dark. For a loan pricing model, testers might simulate a 2008-style unemployment spike and check whether the model's outputs stay within sane bounds rather than producing nonsensical approvals or blanket rejections.

For generative AI tools (a chatbot handling customer disputes, an LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.Voir la définition complète → drafting credit memos), stress testing includes adversarial prompting: red teams try to make the system leak confidential data, hallucinate a false account balance, or produce advice that sounds like it's giving investment recommendations without a license.

Step 3: Bias and fair-lending checks

US fair lending law (the Equal Credit Opportunity Act, ECOA, enforced by the CFPB and prudential regulators) prohibits discrimination on protected characteristics like race, sex, and age, and courts have applied it to model outcomes, not just model intent. A model can be legally blind to race and still produce racially disparate outcomes if it uses correlated proxies like zip code.

Standard pre-deployment checks include:

• Disparate impact testing: comparing approval rates, pricing, or fraud-flag rates across demographic groups, often using the "80% rule" (a group's approval rate should be at least 80% of the highest-approved group's rate) as a screening trigger, not a legal safe harbor.
• Proxy variable review: checking whether seemingly neutral features (zip code, device type, education level) act as stand-ins for protected characteristics.
• Explainability sign-off: for credit decisions, US law (ECOA's Regulation B) requires banks give applicants specific adverse action reasons. If a model can't produce a human-readable reason, it cannot go live for that use case, regardless of accuracy.

Step 4: Documentation and model cards

Every model entering production needs a model card: a standardized document covering intended use, training data provenancedata provenanceData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète →, known limitations, performance across subgroups, and who owns it. This isn't bureaucratic theater; it's what lets an auditor, a regulator, or next year's risk team understand a decision they didn't make.

Google's original model card framework has become an informal industry template, though banks add sections specific to SR 11-7: validation status, approval date, and re-review trigger conditions.

Step 5: Human-in-the-loop and override design

No high-stakes model deploys without a defined human checkpoint. The design question is *where*: before the decision (human approves every output), after (human reviews a sample), or only on exceptions (human reviews only low-confidence or high-value cases).

For a mortgage underwriting model, regulators generally expect human review of denials and high-value approvals. For a marketing next-best-offer model, a light-touch sampling review is proportionate. Getting this proportionality wrong in either direction (over-checking low-risk decisions, under-checking high-risk ones) is a common finding in supervisory exams.

Step 6: Kill-switch design

A kill switch is a pre-built mechanism to instantly revert to the prior model or a manual process, without an emergency engineering scramble. Good kill-switch design means:

• A named, on-call decision-maker with authority to pull the trigger (not a committee that needs to convene).
• A tested rollback path, meaning the old model or manual process still runs in parallel or can be reactivated in minutes, not days.
• Pre-agreed trigger thresholds: e.g., "if false-positive rate exceeds X% for 2 consecutive hours, or if the model flags an unexplainable spike in a protected-class subgroup, escalate immediately."

Banks that skip this step often discover, mid-incident, that reverting requires redeploying infrastructure that takes 48 hours, an unacceptable window when a broken model is actively denying loans or missing fraud.

Step 7: Sign-off and staged rollout

Final deployment typically follows a staged pattern: internal employees first, then a small customer segment (often 1 to 5%), then gradual expansion while monitoring dashboards for drift. Sign-off requires distinct approvals from model risk management, compliance, and the business line, each testing a different question: is it statistically sound, is it legal, and does it work for the customer.

🎬 [VIDEO: "Model Risk Management Explained" - youtube.com - search for recent bank risk-management explainer channels covering SR 11-7 and model validation lifecycles for a visual walkthrough of this governance chain]

A lightweight pseudocode view of a rollout monitor:

if live_false_positive_rate > baseline * 1.3:
    alert_risk_team()
if subgroup_approval_gap > 0.20:  # 80% rule breach
    pause_rollout()
if confidence_score < threshold:
    route_to_human_reviewer()

Simple rules like these, wired into production monitoring, are what actually catch drift in week one rather than in next quarter's audit.

Key Takeaways

• The pre-deployment gauntlet exists because AI model risk is regulated (SR 11-7 in the US, EU AI Act's conformity assessments in Europe), not optional best practice.
• Champion-challenger testing and stress scenarios catch performance problems; bias testing and explainability checks catch legal exposure under laws like ECOA.
• Every high-stakes model needs a model card, a defined human checkpoint, and a kill switch with a named owner and a tested rollback path, built before launch, not improvised during an incident.
• Staged rollout with live monitoring thresholds (not just pre-launch testing) is where many real-world problems are actually caught.
• Proportionality matters: the intensity of each check should scale with the stakes of the decision, over-checking low-risk models wastes resources, under-checking high-risk ones invites the failures regulators actually punish.