Where AI models quietly break in production

3. Silent degradation. A model's performance decays gradually, not with a crash. A fraud-scoring model loses precision over 18 months as fraud tactics evolve (new staged-accident schemes, synthetic identities), but nobody notices because the model still returns scores and claims still get processed. No alert fires because nothing "breaks" in an IT sense.

A working taxonomy of model risk

Group these failures into four buckets, borrowed and adapted from model risk management (MRM) frameworks used in banking (see the Federal Reserve's SR 11-7 guidance, the reference standard even outside banking):

| Risk type | What it looks like in insurance | Detection signal |

|---|---|---|

| Data risk | Feature drift post-catastrophe, stale credit data, sensor/telematics gaps | Distribution shift in input features vs. training baseline |

| Fairness/proxy risk | Credit score, ZIP code, or occupation reconstructing protected-class signal | Disparate impact ratio, subgroup outcome audits |

| Performance decay risk | Fraud model losing precision as tactics evolve; churn model stale after a product change | Rolling precision/recall vs. time, calibration drift |

| Governance/process risk | No documented owner, no retraining trigger, vendor model treated as black box | Absence of a model inventory entry or review cadence |

Each bucket needs a different check. Accuracy metrics alone only catch (partially) bucket 3, and only after damage is done.

Concrete example: proxy discrimination in credit-based scoring

Credit-based insurance scores are widely used in US auto and home underwriting, banned or restricted in some states (California, Massachusetts, Hawaii prohibit their use in auto insurance pricing, as of recent state law; check current state-by-state status since this shifts). The mechanism:

• Credit history correlates with income, and income correlates with race due to historical and structural factors unrelated to driving risk.
• A model trained to minimize loss ratio will pick up the credit signal because it's predictive, without "knowing" it's a proxy.
• Result: a facially neutral variable produces a disparate impact on protected groups, actionable under fair lending analogues applied to insurance by state regulators and, in principle, under EU non-discrimination law.

A simple worked check: the four-fifths rule (borrowed from US employment discrimination law, EEOC guidance) as a first screen.

This is a screening heuristic, not a legal safe harbor. It tells you where to look, not whether you're compliant. A ratio of 0.78 in a pricing tier should trigger a full subgroup audit before deployment, not a pass/fail verdict.

The catastrophe drift problem, mechanically

Why does a fraud model break specifically after a hurricane? Because its training data encodes a world where:

• High claim velocity in a short window = suspicious
• Multiple claims from the same geographic cluster = suspicious
• Claims filed by public adjusters in bulk = suspicious

After Ian, Ida, or any major catastrophe, all three become *normal*. The model has no concept of "declared disaster zone" unless someone explicitly engineered that feature and retrained.

The checks that catch this before deployment or before it does damage:

1. Scenario stress-testing. Before go-live, run the model against a synthetic post-catastrophe claim distribution, not just historical validation data.

2. Feature monitoring with drift alarms. Track the statistical distribution (not just the mean) of key inputs weekly; alarm when it moves outside a defined tolerance (e.g., population stability index above 0.25).

3. A "disaster mode" override. Many mature insurers now build an explicit catastrophe flag that changes model behavior or routes claims to human review during declared events, rather than trusting the base model.

Guardrails before deployment: a practical checklist

Model risk management in insurance is converging on a few non-negotiable pre-deployment gates, echoed in NAIC's model governance principles and the EU AI Act's requirements for high-risk systems (documentation, human oversight, logging, accuracy/robustness testing):

• Model inventory entry: every model in production has a named owner, a documented purpose, and a review date. No exceptions for vendor/black-box models (this is where many insurers using third-party fraud or telematics scoring tools have gaps).
• Subgroup performance audit: test accuracy, false positive rate, and approval rate broken out by protected and quasi-protected classes, not just in aggregate.
• Drift monitoring plan: define what "normal" input distribution looks like, and what triggers a retraining or human review.
• Explainability artifact: a claims adjuster or regulator can get a plain-language reason for a given score or decision (relevant to EU AI Act transparency obligations and to state-level adverse action notice requirements in the US, which long predate AI and originate in fair credit practices law).
• Kill switch / rollback plan: a documented, tested way to revert to the prior model or a manual process if the new model misbehaves post-launch.

None of this replaces actuarial or accuracy testing. It sits alongside it.

🎬 [VIDEO: "Algorithmic Bias in Insurance Pricing" - youtube.com/results?search_query=algorithmic+bias+insurance+pricing+NAIC - search for recent NAIC or insurance-industry panel discussions on proxy discrimination in underwriting models, a useful visual walkthrough of how proxies form]

Key Takeaways

• Accuracy metrics measure the past, not production risk. A model can pass validation and still fail in deployment due to feature drift, proxy discrimination, or silent decay, none of which a single accuracy number reveals.
• Feature drift after catastrophe events is a known, recurring failure pattern in fraud and claims models; the fix is scenario stress-testing plus explicit "disaster mode" handling, not just retraining after the fact.
• Proxy discrimination hides in seemingly neutral variables like credit-based scores or ZIP codes; the four-fifths rule is a useful screening heuristic, not a compliance guarantee, so deeper subgroup audits are required.
• Silent degradation has no crash signal. It requires active rolling performance monitoring (precision, recall, calibration over time), because nothing alerts you by default.
• Governance is a deployment gate, not paperwork: model inventory, subgroup audits, drift monitoring, explainability, and rollback plans should all exist before go-live, aligned with both NAIC guidance in the US and the EU AI Act's high-risk system requirements.