# The AI risk taxonomy every banker needs beyond bias and hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition →
In August 2024, several large asset managers' trading algorithms reportedly reacted to the same macro signal within minutes of each other, amplifying a sharp selloff in Japanese equities and the yen carry trade unwind. No single model was "wrong." Each behaved exactly as designed. The danger was that thousands of institutions were, in effect, running variations of the same model, trained on the same data, reacting the same way, at the same time. That is not a bias problem or a hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → problem. It is a systemic AI risk, and most bank risk committees don't have a name for it on their checklist yet.
This lesson gives you four names regulators and supervisors actually use: concentration risk, third-party contagion, feedback loops, and model herding. Knowing the vocabulary is table stakes for any governance conversation in banking today.
Bias (systematic unfairness in model outputs, often across protected groups) and hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → (a generative model producing confident but false content) dominate public discussion because they're visible and easy to demonstrate with a single bad output.
But banking supervisors, including the Federal Reserve, the Bank of England's Prudential Regulation Authority (PRA), and the European Central Bank (ECB), worry more about risks that emerge at the *system* level: many banks using similar models, similar vendors, or similar data, creating correlated failure modes that no single institution's model validation would catch. This is the domain of model risk management, formalized in the US under SR 11-7, the Fed's foundational guidance on model risk, and increasingly extended to AI/ML models specifically.
Concentration risk here means dependence on a small number of AI providers, foundation models, or datasets across the sector.
Concrete example: multiple mid-size US banks use the same third-party vendor for AI-driven credit scoring built on a similar base model (say, a fine-tuned version of a major foundation model from OpenAI, Anthropic, or Google). If that vendor's underlying model has a systematic flaw, an outdated macro assumption, a data drift issue, dozens of banks inherit the same blind spot simultaneously.
This mirrors the "too many eggs, one basket" problem regulators know well from cloud infrastructure. The Bank of England and the ECB have flagged that the shift to a handful of hyperscale cloud and AI providers (Microsoft Azure, AWS, Google Cloud) concentrates operational risk across a large share of the financial sector. The EU's Digital Operational Resilience Act (DORA), which became fully applicable in January 2025, specifically requires banks to mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → and report critical third-party ICT (information and communication technology) providers for exactly this reason.
Worked example: if 40% of European retail banks use the same three cloud-hosted AI vendors for fraud detection, a shared vulnerability or model update error doesn't cause one bank's fraud losses to rise. It can cause correlated fraud detection failures across a meaningful share of the retail banking sector simultaneously.
Third-party contagion is concentration risk's cousin: it's about how a failure *propagates*, not just where dependency sits.
A lending desk example: a bank outsources income verification to an AI document-processing vendor. The vendor pushes a model update that misreads a common payslip format, silently approving loans it should flag. The bank doesn't know until weeks later, when delinquency data looks odd. By then, thousands of loans are affected. The bank owns the credit risk; the vendor owned the model.
This is why DORA and, in the US, the OCC's (Office of the Comptroller of the Currency) third-party risk management guidance both push banks toward continuous monitoring of vendor AI performance, not just onboarding due diligence. Governance has to extend past the bank's own walls.
A feedback loop occurs when a model's outputs alter the future data it will be trained or evaluated on, reinforcing (or distorting) its own predictions.
Trading desk example: a market-making algorithm systematically widens spreads for a certain client segment based on predicted "toxic flow" (informed trading likely to be profitable at the market maker's expense). Those clients get worse execution, trade less through that desk, and the training data increasingly reflects only the "safe" client behavior the model already expected. The model looks more accurate over time, but only because it shaped the population it observes.
Lending desk example: a credit model denies applicants below a certain score. Those applicants never get a loan, so the bank never observes whether they'd have repaid. The model's "accuracy" is measured only on the population it approved, a form of selection bias that compounds with each retraining cycle. This is sometimes called the "selective labels problem" in credit modeling research, well documented in academic literature including work summarized by the Consumer Financial Protection Bureau.
Model herding is when independently built models, because they're trained on similar data or similar market signals, converge on similar decisions, especially under stress, amplifying moves that a diverse set of human or model decision-makers would have dampened.
This is the mechanism behind the opening example. It's structurally similar to the risk regulators identified after the 1987 portfolio insurance crash and the 2010 Flash Crash, except AI-driven trading strategies raise the stakes because more of the "thinking" is now automated and models can be retrained faster than human strategies adapt.
The Bank for International Settlements (BIS) and the Financial Stability Board (FSB) have both published on this specifically. The FSB's 2024 report on AI and financial stability flags herding and correlated risk-taking as a top-tier systemic concern precisely because it doesn't show up in any single institution's risk dashboard.
Herding risk checklist (desk-level):
1. Do we know which foundation model(s) our vendors' tools are built on?
2. Would a competitor's model likely reach the same decision on the same data?
3. If yes to #2, what % of the market shares this dependency? (estimate)
4. Do we have a "diversity of models" requirement in procurement policy?
5. Is there a manual override / circuit breaker if signals move in unison?If you can't answer question 1 for your top three AI vendors, that's a governance gap worth raising, not a hypothetical.
Knowledge check
1. Why did the August 2024 market episode described in the lesson illustrate a systemic AI risk rather than a bias or hallucination problem?
2. Why do banking supervisors like the Federal Reserve, PRA, and ECB emphasize risks beyond bias and hallucination?
3. A risk committee is evaluating whether their bank's heavy reliance on a single dominant foundation model provider, also used by most competitors, poses a governance concern. Which risk category does this best represent?
4. Select ALL correct answers about why bias and hallucination dominate public discussion of AI risk in banking, according to the lesson.
Select all the correct answers.
5. Select ALL correct answers describing characteristics of systemic AI risks like model herding or concentration risk, as distinct from bias or hallucination.
Select all the correct answers.
Given this taxonomy, a pre-deployment checklist for any AI system in a bank should include:
🎬 [VIDEO: "How AI Could Cause the Next Financial Crisis" - youtube.com/@TheEconomist - a concise explainer on systemic AI risk in finance, covering herding and concentration dynamics relevant to this lesson]