# Model risk management for investment AI
In 2007, a cluster of quantitative equity funds unwound within days. Their models, trained on similar data with similar assumptions, all sold the same names at the same time. Nobody had modeled the risk that the models themselves were the crowd. Almost two decades later, machine learning has made that failure mode faster, more opaque, and harder to detect. This lesson shows you how to govern it.
SR 11-7 is guidance issued in 2011 by the US Federal Reserve and the Office of the Comptroller of the Currency (OCC). It defines a "model" as any quantitative method that turns input data into an estimate, and it demands that every model be validated independently, documented fully, and monitored continuously. It was written for banks, but it has become the de facto global standard for model risk discipline.
Read the primary source once. It is short and readable: Federal Reserve SR 11-7 guidance.
The core idea: model risk is the risk of loss from a model that is wrong or used wrongly. SR 11-7 breaks it into two sources.
ML alpha models fail on both counts more often than linear models did, because they are more flexible and less interpretable. Flexibility is exactly what lets them memorize noise.
SR 11-7 rests on three pillars. Here is how each maps to an AI-driven investment process.
Every ML signal needs a written "model card": what it predicts, on what data, over what universe, with what known limits. If a portfolio manager cannot explain in one paragraph what a signal does, it should not size a position.
For asset management specifically, document the look-ahead discipline. A look-ahead error means the model saw information it would not have had in real time. Example: using a company's restated earnings (published months later) as if available on the original report date. This single error is the most common cause of backtests that look brilliant and trade like garbage.
The team that builds the model cannot be the team that signs off on it. In practice, a mid-size asset manager runs a validation function that re-implements the signal from scratch, on held-out data, and tries to break it. This is not a formality. Independent validation catches the subtle leakage the builders are blind to because they want the model to work.
Markets are non-stationary. A signal that worked in the low-rate regime of 2015 to 2021 may be actively harmful in a higher-rate, higher-inflation regime. Monitoring means tracking live performance against backtested expectations and triggering review when they diverge.
Test enough strategies against history and some will look profitable by pure chance. This is backtest overfitting: mistaking luck for skill.
The math is unforgiving. Marcos Lopez de Prado has shown that if you try many strategy configurations and keep the best Sharpe ratio, that best result is biased upward, sometimes wildly. The Deflated Sharpe Ratio adjusts a reported Sharpe for the number of trials run and the length of the track record.
A simple, concrete guardrail: log every backtest you run.
# Track the number of independent trials so you can deflate later
import json, datetime
def log_trial(strategy_id, config, sharpe):
record = {
"ts": datetime.datetime.utcnow().isoformat(),
"strategy_id": strategy_id,
"config": config,
"in_sample_sharpe": sharpe,
}
with open("trials.log", "a") as f:
f.write(json.dumps(record) + "\n")
# A reported Sharpe of 2.0 from 1 trial is very different
# from a Sharpe of 2.0 selected as best out of 500 trials.Worked example (illustrative, not a real strategy): suppose you test 100 signal variants and the best shows an annual Sharpe of 1.5 over 5 years. Because you selected the maximum of 100 noisy estimates, a meaningful chunk of that 1.5 is selection luck. Deflation might cut the "true" expected Sharpe toward 0.5 or below. The rule of thumb: the more configurations you tried, the higher the bar the winner must clear.
An ML model learns the joint distribution of its training data. When that distribution changes (a regime shift), the model does not know it is now extrapolating.
Guardrails:
When a gradient-boosted model or neural net produces a trade, "the model said so" is not an acceptable answer to a risk committee.
Use explainability tools, most commonly SHAP (SHapley Additive exPlanations), which attributes a prediction to its input features. If a stock-selection model's top signal is suddenly one obscure feature, that is a flag. Explainability does not make a black box safe. It makes it auditable, which is what governance requires.
Model risk in investment AI now sits at the intersection of several real regimes.
The common thread across all of them: accountability must attach to a named human, and decisions must be documented and reproducible.
Knowledge check
1. The 2007 quant fund unwind is used to illustrate which model risk that is especially relevant to ML-driven investing?
2. According to SR 11-7's definition, which of the following would count as a 'model' requiring governance?
3. Why does the lesson argue that ML alpha models tend to fail on both SR 11-7 risk sources more often than linear models?
4. Select ALL correct answers about the two sources of model risk identified by SR 11-7.
Select all the correct answers.
5. Select ALL correct answers about what SR 11-7's pillar of 'robust development and documentation' requires when applied to an ML alpha signal.
Select all the correct answers.
Before an ML signal touches real capital, walk this gate. Fail any item, do not deploy.
Data integrity
Overfitting control
Robustness
Governance
That last point matters most. The 2007 quant unwind was not caused by one bad model. It was caused by no one having the authority or the trigger to step out of the crowd in time.