The pre-deployment checklist that stands up to an audit

# The pre-deployment checklist that stands up to an audit

A chief actuary at a mid-sized US personal auto carrier is asked to sign an attestation that a new AI-based underwriting model is fair, explainable, and safe to launch. She has three days. The state insurance regulator has already signaled it will request the full model file if loss ratios move unexpectedly in the first two quarters. This is not hypothetical: it is roughly the position chief actuaries have been in since the NAIC (National Association of Insurance Commissioners) adopted its Model Bulletin on the Use of Artificial Intelligence Systems by Insurers in 2023, now adopted by most US states.

This lesson walks through the checklist, in order, that a defensible sign-off actually requires.

Why "it works" is not a sign-off

A model that predicts loss ratio well can still fail governance. Three failure modes matter most in underwriting and pricing:

• Proxy discrimination: a variable correlates with a protected class (race, religion, in some states sexual orientation) even though the class itself is excluded. Credit-based insurance scores have faced this criticism for years.
• Silent drift: the model was validated on 2023 data; by 2026, claims patterns shifted (inflation in auto repair costs, new weather patterns for property) and the model quietly misprices risk.
• Black-box liability: if a regulator or policyholder asks "why was I denied or upcharged," and no one in the company can answer in plain language, that alone can trigger action under state unfair trade practices laws.

The checklist below is built to close all three gaps before go-live.

Step 1: Disparate impact testing

Disparate impact means a facially neutral variable produces materially worse outcomes for a protected group, regardless of intent. This is the first test run, because if it fails, nothing else matters yet.

Standard practice, drawn from methods used in US fair lending and adapted by state regulators (Colorado's SB21-169 algorithm testing rules are the most codified example as of 2025):

1. Score the model on a holdout dataset with demographic attributes attached (collected separately, never fed into the model itself).

2. Compare outcomes across groups using an adverse impact ratio: the approval/pricing-favorable rate for the protected group divided by the rate for the reference group.

3. A commonly cited threshold (borrowed from the US EEOC's "four-fifths rule" in employment, adapted informally in insurance testing) flags ratios below 0.80 for further review. This is a screening heuristic, not a legal safe harbor.

Worked example: Model approves preferred-tier pricing for 60% of one demographic group and 42% of another.

Adverse impact ratio = 42 / 60 = 0.70 → below the 0.80 screening line → triggers deeper review of which variables are driving the gap.

Colorado's rules require this kind of testing specifically for life insurance algorithms as a live regulatory requirement; other lines and states are following. See the NAIC's AI governance resource center for the state-by-state state of play.

Step 2: Proxy variable audit

Passing the aggregate disparate impact test is not enough. The chief actuary needs a variable-by-variable audit:

• Run correlation analysis between every model input and known protected-class proxies (ZIP code correlating with race; occupation correlating with religion in some populations).
• Test removal sensitivity: drop the suspect variable, retrain, see if the disparate impact ratio improves without materially hurting predictive accuracy (measured by Gini coefficient or AUC). If accuracy barely moves but fairness improves, that variable is a weak signal riding on a proxy, and is usually removed.
• Document the decision either way. Regulators care less about the outcome than about proof the analysis happened.

Step 3: Explainability documentation

Explainability here means the ability to state, for any individual decision, which factors drove it and by how much.

Two levels are expected in a mature governance file:

• Global explainability: which variables matter most across the whole book (feature importance rankings, typically via SHAP, Shapley Additive Explanations, values).
• Local explainability: for any single policyholder, a plain-language reason code ("higher premium driven primarily by vehicle repair cost index and prior claims frequency").

A minimal SHAP-based reason code snippet looks like this:

import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer(X_applicant)

# top 3 drivers for this specific applicant
top_features = (
    shap_values[0].values
    .argsort()[-3:][::-1]
)
reason_codes = [feature_names[i] for i in top_features]

This output feeds directly into adverse action notices, which most US states require under unfair claims and trade practices statutes when a policyholder is denied, non-renewed, or charged a higher rate.

Step 4: Challenger-model backtesting

Before go-live, the incumbent (current production) model and the challenger (new AI model) are run side by side on the same historical book, out-of-time.

Checklist items:

• Out-of-time validation: test on a period the model never saw in training (e.g., train through 2023, test on 2024 to 2025 claims).
• Loss ratio stability by segment: check performance isn't only good in aggregate but stable across state, vehicle class, age band.
• Champion-challenger live pilot: run the challenger in shadow mode, scoring real applications without acting on the score, for a defined window (often one full underwriting cycle, commonly a quarter) before full cutover.
• Override rate check: track how often underwriters override the model. A sudden spike after deployment is an early warning sign of miscalibration or user distrust, both worth investigating.

For property and casualty lines, this backtesting discipline mirrors what the International Association of Insurance Supervisors (IAIS) describes in its supervisory guidance on AI, and increasingly maps to the EU's AI Act (Regulation (EU) 2024/1689) classification of certain insurance pricing and underwriting systems as "high-risk," which mandates documented testing, human oversight, and post-market monitoring for insurers operating in the EU.

Step 5: Governance sign-off package

By the time the chief actuary is asked to sign, the package should contain:

1. Disparate impact test results with the adverse impact ratios by protected class.

2. Proxy variable audit with retained/removed decisions and rationale.

3. Explainability documentation: global feature importance plus sample local reason codes.

4. Challenger backtest results, including out-of-time performance and shadow-mode pilot data.

5. A monitoring plan: named owner, review cadence (commonly quarterly), and defined triggers for re-review (e.g., loss ratio deviation beyond a set threshold, or adverse impact ratio drifting below 0.80 post-launch).

6. Model risk classification under the company's internal model risk management framework, consistent with the Federal Reserve/OCC's SR 11-7 model risk guidance, originally written for banks but widely adopted as best practice by insurance model risk teams.

Without item 5, the sign-off is a snapshot, not governance. Models drift; the checklist has to repeat, not just happen once.

🎬 [VIDEO: "How Insurers Are Using AI Responsibly" - youtube.com - search for NAIC or Society of Actuaries panel discussions on AI governance in insurance underwriting, useful for seeing regulators and actuaries debate these checks in their own words]

Key Takeaways

• Disparate impact testing comes first: compute adverse impact ratios by protected class before evaluating anything else; a common screening threshold is 0.80, borrowed from US employment law and adapted informally in insurance.
• A passing aggregate score is not enough: audit individual variables for proxy discrimination (ZIP code, occupation) using removal sensitivity tests.
• Explainability must work at two levels: global feature importance for regulators, and local, plain-language reason codes for individual policyholders (often required for adverse action notices).