Cataloguing the AI risk taxonomy

In 2024, an Air Canada chatbot invented a refund policy that did not exist, and a tribunal held the airline liable for what its AI said. Now transpose that to a wealth manager whose LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → copilot tells a client that a structured note is "capital protected" when it is not. Same failure mode, higher stakes, and a regulator (the SEC or the FCA) watching. This lesson turns fuzzy AI anxiety into a concrete, scored risk register you could hand to a Chief Risk Officer tomorrow.

*LLM = Large Language Model, the technology behind chatbots like ChatGPT. Copilot = an AI assistant embedded in a workflow, for example drafting client emails or summarising research.*

Why a taxonomy, not a worry list

A taxonomy forces you to name each risk, locate where it lives in your book, and score it consistently. Regulators now expect this. The EU AI Act (in force since 2024, with obligations phasing in through 2026 and 2027) requires risk classification for AI systems. In the US, the SEC has pursued "AI washing" (overstating AI capabilities to investors) and the NIST AI Risk Management Framework gives a free, vendor-neutral structure many US firms adopt voluntarily.

*Book = the portfolio and client base a firm manages. A "live" book means real client money, not a sandbox.*

We will score each risk on two axes, 1 to 5:

• Likelihood: how often this could plausibly happen.
• Impact: client harm, regulatory exposure, and financial loss combined.

Risk score = Likelihood x Impact. Anything scoring 15 or above needs a named owner and a mitigation before deployment.

The four headline risks

1. Hallucinating LLM copilots

A hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → is when an LLM produces fluent, confident text that is factually wrong. In asset management this shows up in client-facing chat, RFP (Request For Proposal) drafting, and research summarisation.

Concrete scene: a private bank deploys a copilot to summarise fund fact sheets. It states a fund's ongoing charges figure as 0.45% when the real figure is 0.95%. A relationship manager pastes this into a client email. That is a mis-selling exposure under MiFID II (the EU's Markets in Financial Instruments Directive, which governs suitability and disclosure).

• Likelihood: 4 (hallucination is inherent to current LLMs)
• Impact: 4 (direct client harm, regulatory breach)
• Score: 16

GuardrailGuardrailRules and controls that keep an AI system inside safe, legal and on-brand boundaries, blocking outputs and actions that cross the line.View full definition →: retrieval-augmented generationretrieval-augmented generationA method that lets an AI model answer using your own documents, retrieving relevant passages before generating a response instead of relying only on training data.View full definition → (RAG), where the model must cite a source document, plus a rule that any number quoted must link back to the source line. Never let a copilot generate figures freely.

2. Correlated model herding

This is the quiet, systemic one. If many managers license the same foundation model or the same signal vendor, their portfolios can start reacting to the same prompts in the same way. When everyone's AI says "reduce duration" at once, you get crowded trades and liquidity gaps.

The Bank of England and the Financial Stability Board have both flagged this concentration risk. It is not hypothetical: quant strategies have shown crowding before (the August 2007 "quant quake" is the classic pre-AI example).

• Likelihood: 3 (rising as adoption concentrates)
• Impact: 5 (systemic, hits the whole book in stressed markets)
• Score: 15

Guardrail: track vendor and model diversity as a monitored metric. Stress test the book assuming a correlated AI-driven signal fires across your peer group simultaneously.

3. Data leakage

Two flavours. First, **confidential data (client PII, positions, deal information) leaking *into* a public model when staff paste it into a consumer chatbot**. Second, model outputs leaking information they should not, for example a copilot trained on one client's mandate surfacing details to another.

*PII = Personally Identifiable Information, protected under GDPR in Europe and various state laws in the US.*

Concrete scene: an analyst pastes a pre-announcement M&A position into a public LLM to "help write the memo." That data may now sit on a third-party server. Under GDPR, fines can reach 4% of global annual turnover. Under market abuse rules (the EU Market Abuse Regulation, MAR), the position itself is inside information.

• Likelihood: 4 (human behaviour is the weak link)
• Impact: 5 (regulatory, reputational, potentially criminal)
• Score: 20

Guardrail: block consumer AI tools on corporate devices, provide an enterprise instance with contractual data protection (no training on your data), and log every prompt.

4. Vendor concentration

Most firms do not build foundation models. They rent them from a handful of providers (OpenAI, Anthropic, Google, Microsoft, plus cloud hosts AWS and Azure). If one provider changes pricing, deprecates a model version, or suffers an outage, your workflow breaks.

Concrete scene: a model version your compliance workflow depends on is retired with 90 days notice, and the replacement behaves differently on your prompts. You must re-validate everything.

• Likelihood: 3
• Impact: 4 (operational, not immediately client-facing)
• Score: 12

Guardrail: multi-vendor abstraction (route prompts through a layer that can switch providers), and contractual notice periods. This is classic operational resilience, now covered in Europe by DORA (the Digital Operational Resilience Act, applying to financial entities since January 2025), which explicitly captures critical ICT third parties.

The risk register, assembled

Everything at 15 or above (three of four) needs mitigation before go-live. Data leakage tops the list not because it is exotic but because human behaviour drives the likelihood up.

🎬 [VIDEO: "The Air Canada Chatbot Case" - youtube.com - a short explainer on the tribunal decision holding a company liable for its AI's false statements, directly relevant to copilot liability]

Scoring in code

If you maintain this register in a spreadsheet or a data pipelinedata pipelineETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.View full definition →, the logic is trivial and worth automating so scores update as your estimates change:

risks = [
    {"name": "Data leakage",        "likelihood": 4, "impact": 5},
    {"name": "Hallucinating copilot","likelihood": 4, "impact": 4},
    {"name": "Model herding",       "likelihood": 3, "impact": 5},
    {"name": "Vendor concentration","likelihood": 3, "impact": 4},
]

THRESHOLD = 15
for r in risks:
    r["score"] = r["likelihood"] * r["impact"]
    r["action"] = "MITIGATE PRE-DEPLOY" if r["score"] >= THRESHOLD else "monitor"

for r in sorted(risks, key=lambda x: -x["score"]):
    print(f'{r["name"]:24} {r["score"]:>2}  {r["action"]}')

The point is not the code. It is that the threshold is explicit, auditable, and consistent across every AI use case you review.

From register to guardrails

A score is a diagnosis. The guardrail is the treatment. MapMapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → each high-scoring risk to a check you run *before* deployment and a control you run *continuously* after.

Pre-deployment checks

• Model validationModel validationThe process of measuring whether an AI model performs well enough, and safely enough, for the business decision it is meant to support.View full definition →: test the copilot against a fixed set of known-answer questions. Measure the hallucination rate. If it invents figures more than a tiny fraction of the time, it does not go client-facing.
• Red teaming: deliberately try to make the model leak data or say something non-compliant. Document what breaks.
• Human in the loop: for any output that reaches a client or triggers a trade, define who signs off. The EU AI Act mandates human oversight for higher-risk systems.

Continuous controls

• Prompt and output logging: keep an auditable trail. When a regulator asks "what did your AI tell this client," you must answer.
• Drift monitoring: model behaviour changes when the vendor updates it. Re-run your known-answer tests on a schedule.
• Concentration dashboard: track how much of your workflow depends on a single model or vendor.

Who owns this

Governance is not "the AI team's problem." Model risk sits with the CROCROConversion Rate Optimization (CRO) is the systematic practice of increasing the percentage of users who complete a desired action, using data, testing, and user research.View full definition → and CIO, data leakage with the CISO, vendor resilience with the COO, and client-facing accuracy with the head of advice. The board signs the risk appetite. This mirrors the "three lines of defence" model that risk professionals already use: the business owns the risk, risk and compliance challenge it, and internal audit checks the whole thing.

Key takeaways

• Score, do not vibe. Likelihood x Impact, with a published threshold (we used 15), turns AI anxiety into an auditable register with named owners.
• Data leakage usually tops the list because human behaviour drives likelihood. Block consumer AI tools, provide a contractually protected enterprise instance, and log everything.
• Never let a copilot invent numbers. Use retrieval with mandatory source citation, and keep a human sign-off on anything client-facing or trade-triggering.
• Model herding is the systemic sleeper. Monitor vendor and model diversity, and stress test for the scenario where every peer's AI fires the same signal at once.
• The regulation is real and named. EU AI Act, GDPR, MiFID II, MAR, and DORA in Europe; the SEC and the NIST AI RMF in the US. Map each guardrail to the rule it satisfies.