The AI risk taxonomy every banker needs beyond bias and hallucination

# The AI risk taxonomy every banker needs beyond bias and hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.Voir la définition complète →

In August 2024, several large asset managers' trading algorithms reportedly reacted to the same macro signal within minutes of each other, amplifying a sharp selloff in Japanese equities and the yen carry trade unwind. No single model was "wrong." Each behaved exactly as designed. The danger was that thousands of institutions were, in effect, running variations of the same model, trained on the same data, reacting the same way, at the same time. That is not a bias problem or a hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.Voir la définition complète → problem. It is a systemic AI risk, and most bank risk committees don't have a name for it on their checklist yet.

This lesson gives you four names regulators and supervisors actually use: concentration risk, third-party contagion, feedback loops, and model herding. Knowing the vocabulary is table stakes for any governance conversation in banking today.

Why bias and hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.Voir la définition complète → aren't enough

Bias (systematic unfairness in model outputs, often across protected groups) and hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.Voir la définition complète → (a generative model producing confident but false content) dominate public discussion because they're visible and easy to demonstrate with a single bad output.

But banking supervisors, including the Federal Reserve, the Bank of England's Prudential Regulation Authority (PRA), and the European Central Bank (ECB), worry more about risks that emerge at the *system* level: many banks using similar models, similar vendors, or similar data, creating correlated failure modes that no single institution's model validation would catch. This is the domain of model risk management, formalized in the US under SR 11-7, the Fed's foundational guidance on model risk, and increasingly extended to AI/ML models specifically.

Concentration risk: when everyone rents the same brain

Concentration risk here means dependence on a small number of AI providers, foundation models, or datasets across the sector.

Concrete example: multiple mid-size US banks use the same third-party vendor for AI-driven credit scoring built on a similar base model (say, a fine-tuned version of a major foundation model from OpenAI, Anthropic, or Google). If that vendor's underlying model has a systematic flaw, an outdated macro assumption, a data drift issue, dozens of banks inherit the same blind spot simultaneously.

This mirrors the "too many eggs, one basket" problem regulators know well from cloud infrastructure. The Bank of England and the ECB have flagged that the shift to a handful of hyperscale cloud and AI providers (Microsoft Azure, AWS, Google Cloud) concentrates operational risk across a large share of the financial sector. The EU's Digital Operational Resilience Act (DORA), which became fully applicable in January 2025, specifically requires banks to mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.Voir la définition complète → and report critical third-party ICT (information and communication technology) providers for exactly this reason.

Worked example: if 40% of European retail banks use the same three cloud-hosted AI vendors for fraud detection, a shared vulnerability or model update error doesn't cause one bank's fraud losses to rise. It can cause correlated fraud detection failures across a meaningful share of the retail banking sector simultaneously.

Third-party contagion: your vendor's outage is your risk event

Third-party contagion is concentration risk's cousin: it's about how a failure *propagates*, not just where dependency sits.

A lending desk example: a bank outsources income verification to an AI document-processing vendor. The vendor pushes a model update that misreads a common payslip format, silently approving loans it should flag. The bank doesn't know until weeks later, when delinquency data looks odd. By then, thousands of loans are affected. The bank owns the credit risk; the vendor owned the model.

This is why DORA and, in the US, the OCC's (Office of the Comptroller of the Currency) third-party risk management guidance both push banks toward continuous monitoring of vendor AI performance, not just onboarding due diligence. Governance has to extend past the bank's own walls.

Feedback loops: when the model changes the world it predicts

A feedback loop occurs when a model's outputs alter the future data it will be trained or evaluated on, reinforcing (or distorting) its own predictions.

Trading desk example: a market-making algorithm systematically widens spreads for a certain client segment based on predicted "toxic flow" (informed trading likely to be profitable at the market maker's expense). Those clients get worse execution, trade less through that desk, and the training data increasingly reflects only the "safe" client behavior the model already expected. The model looks more accurate over time, but only because it shaped the population it observes.

Lending desk example: a credit model denies applicants below a certain score. Those applicants never get a loan, so the bank never observes whether they'd have repaid. The model's "accuracy" is measured only on the population it approved, a form of selection bias that compounds with each retraining cycle. This is sometimes called the "selective labels problem" in credit modeling research, well documented in academic literature including work summarized by the Consumer Financial Protection Bureau.

Model herding: everyone runs to the same exit

Model herding is when independently built models, because they're trained on similar data or similar market signals, converge on similar decisions, especially under stress, amplifying moves that a diverse set of human or model decision-makers would have dampened.

This is the mechanism behind the opening example. It's structurally similar to the risk regulators identified after the 1987 portfolio insurance crash and the 2010 Flash Crash, except AI-driven trading strategies raise the stakes because more of the "thinking" is now automated and models can be retrained faster than human strategies adapt.

The Bank for International Settlements (BIS) and the Financial Stability Board (FSB) have both published on this specifically. The FSB's 2024 report on AI and financial stability flags herding and correlated risk-taking as a top-tier systemic concern precisely because it doesn't show up in any single institution's risk dashboard.

A simple way to spot herding risk in your own shop

Herding risk checklist (desk-level):
1. Do we know which foundation model(s) our vendors' tools are built on?
2. Would a competitor's model likely reach the same decision on the same data?
3. If yes to #2, what % of the market shares this dependency? (estimate)
4. Do we have a "diversity of models" requirement in procurement policy?
5. Is there a manual override / circuit breaker if signals move in unison?

If you can't answer question 1 for your top three AI vendors, that's a governance gap worth raising, not a hypothetical.

The guardrails: what to check before deployment

Given this taxonomy, a pre-deployment checklist for any AI system in a bank should include:

• Vendor and model lineage mapping: know the foundation model, training data provenancedata provenanceData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète →, and update cadence for every third-party AI tool, per DORA-style requirements.
• Concentration stress testing: model what happens if your top AI vendor has an outage or systematic error, not just what happens if your own model underperforms.
• Feedback loop audits: for any model that influences who gets excluded (loan denials, fraud flags, trading counterparty limits), regularly test performance against a small sample of the excluded population, not just the approved one.
• Correlation checks across the sector: where possible, benchmark your model's decisions against industry consensus; large deviations aren't automatically wrong, but unexamined convergence with everyone else is a red flag under herding risk.
• Human override points: define, in advance, the specific market or portfolio conditions under which automated decisions get paused for human review. SR 11-7 and its AI-specific extensions expect this documented, not improvised during a crisis.