# Spotting model risk before it becomes a loss event
A mid-size US bank ran a collections model that quietly stopped working for eight months. Nobody noticed at first. The model kept scoring delinquent accounts and ranking them for outreach, but the world underneath it had changed: a shift in customer mix after a portfolio acquisition, a new hardship program, a change in how call center agents logged outcomes. The model's predictions drifted away from reality, collectors chased the wrong accounts, recovery rates slipped, and by the time someone reconciled model output against actual roll rates, the bank had absorbed a measurable, avoidable loss. This is a textbook case of model risk realized, not because the model was built badly, but because nobody was watching it after launch.
This lesson is about catching that drift before it shows up in your numbers or in an examiner's letter.
Model risk is the risk of financial loss or bad decisions caused by a model that is wrong, misused, or poorly understood. In the US, the reference framework is the Federal Reserve and OCC's SR 11-7 guidance on model risk management (2011), still the operating standard for banks in 2026. It defines model risk as arising from two sources:
Silent drift, like the collections case, is usually the second one. The model wasn't wrong when it launched. It became wrong when its environment changed and nobody re-checked the fit.
In Europe, the equivalent expectations sit inside the ECB's guide on internal models and, more broadly, the EU AI Act (entered into force 2024, phased obligations through 2026-2027), which classifies creditworthiness and credit-scoring AI systems as "high-risk," triggering mandatory risk management, data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.Voir la définition complète →, and human oversight requirements.
1. Data drift. The statistical properties of incoming data change. Example: a bank acquires a fintech's loan book; the new borrowers have thinner credit files, so the model sees inputs it was never trained on.
2. Concept drift. The relationship between inputs and outcomes changes, even if the input data looks stable. Example: a hardship program launches, so borrowers who look "high risk" by the model's old logic now cure faster because they got payment relief. Same inputs, different outcome.
3. Feedback loop risk. The model's own decisions change future data. Example: a collections model deprioritizes certain accounts, those accounts get less attention, they perform worse, and the model "learns" they were high risk all along, reinforcing a self-fulfilling pattern. This is especially dangerous in underwriting and collections models that influence who gets contacted, approved, or offered relief.
The eight-month gap in the case above existed because the bank monitored *outcomes* (charge-offs, recovery rates) but not *model behavior*. Outcomes lag by months. Behavior signals move faster. Track these:
PSI compares the percentage of accounts in each score bucket now versus at training time.
PSI = Σ (Actual% - Expected%) × ln(Actual% / Expected%)Say the "low risk" score bucket had 40% of accounts at model training and now has 25%:
Bucket contribution = (0.25 - 0.40) × ln(0.25 / 0.40)
= (-0.15) × ln(0.625)
= (-0.15) × (-0.47)
= 0.0705Sum this across all buckets. If the total exceeds your institution's threshold (commonly 0.1 for "watch" and 0.25 for "act," as estimates), that is a trigger for review, independent of whether losses have shown up yet.
import numpy as np
def psi(expected, actual):
return np.sum((actual - expected) * np.log(actual / expected))
expected = np.array([0.40, 0.35, 0.25]) # training distribution
actual = np.array([0.25, 0.40, 0.35]) # current distribution
print(round(psi(expected, actual), 3))Model risk management (MRM) under SR 11-7 rests on three pillars, and each one maps to a concrete pre-deployment check:
1. Development and documentation: Is there a model card or equivalent record of intended use, data sources, and known limitations? If the collections model's documentation had specified "not validated for hardship-program populations," the drift would have been flagged the day the program launched.
2. Independent validation: A team separate from model developers must test the model before and periodically after launch. This is not optional under US bank supervisory expectations, and under the EU AI Act, high-risk systems require documented conformity assessments before market placement.
3. Ongoing monitoring: Pre-agreed triggers (like the PSI thresholds above) that force a review, not just an annual check-in. Monitoring cadence should match how fast the underlying population can change: monthly or even weekly for collections and fraud models, versus quarterly for slower-moving models like long-term credit risk ratings.
A practical pre-launch checklist:
Vérification des acquis
1. In the collections model case, what was the fundamental cause of the loss event?
2. Under the SR 11-7 framework, how is 'misuse' as a source of model risk best distinguished from a 'fundamental error'?
3. A bank's credit risk model performed well at validation two years ago. Since then, the bank acquired a new loan portfolio with different borrower characteristics, but the model has not been re-validated. What does this scenario primarily illustrate?
4. Select ALL correct answers about why the collections model's failure went undetected for eight months.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about effective practices for catching model risk before it produces a loss event.
Sélectionnez toutes les réponses correctes.
Banks are good at validating models before launch and comparatively weak at watching them after. A 2023 Federal Reserve outlook and industry surveys (see the Fed's supervisory guidance library) repeatedly cite "insufficient ongoing monitoring" as the most common MRM gap examiners find. The pattern is structural: development budgets are visible and funded, monitoring budgets are not.
The collections case reflects this exactly. The model passed validation. Nobody owned it afterward. The bank found out through a quarterly loss review, the slowest possible signal, instead of a weekly PSI dashboard, the fastest one.
🎬 [VIDEO: "Model Risk Management Explained" - https://www.youtube.com/results?search_query=model+risk+management+explained+banking - a walkthrough of SR 11-7 style model risk governance concepts for banking practitioners]