AI risk & the EU AI Act

# AI risk and the EU AI act

In 2023, a Dutch tax authority scandal, the *toeslagenaffaire*, forced the resignation of an entire government cabinet. The trigger: an algorithm that flagged families for childcare-benefit fraud, disproportionately targeting dual-nationality households, and clawed back money from tens of thousands of innocent people. No CDO signed off on a "high-risk AI system." They signed off on a fraud-detection model. That gap, between how the business labels a system and how a regulator will, is exactly the terrain the EU AI Act now maps. Your job is not to memorize the Act. It is to build a triage function that tells you, on any Monday, which of your 200 models could become tomorrow's *toeslagenaffaire* and which are boring enough to leave alone.

The core reframe: risk lives in the use case, not the model

The single most expensive mistake a CDO makes with the AI Act is governing at the wrong altitude. Teams instinctively ask "is this model risky?" The Act asks a different question: "what is this system *used for*, and on whom?"

The same gradient-boosted classifier is unregulated when it ranks marketing leads and high-risk when it screens job applicants. The technology is identical. The risk tier is determined entirely by deployment context. This is why model-centric governance fails, you cannot inspect a .pkl file and know its regulatory exposure. You have to trace it to a decision that affects a human's access to employment, credit, education, essential services, or liberty.

The Act sorts systems into four tiers, and your operating model should mirror them precisely:

• Unacceptable risk (prohibited): social scoring by public authorities, real-time remote biometric identification in public spaces (with narrow exceptions), emotion recognition in workplaces and schools, manipulative subliminal techniques. These are not "govern carefully", they are "do not build." As of February 2025, these prohibitions are already in force.
• High risk: systems used in employment, creditworthiness, essential public and private services, education, critical infrastructure, law enforcement, migration, and administration of justice. Also products or safety components already covered by EU product-safety law. This tier carries the full weight of the Act's obligations.
• Limited risk: systems that interact with humans (chatbots) or generate synthetic content. The obligation is essentially transparency, tell people they're dealing with AI, label deepfakes.
• Minimal risk: everything else, spam filters, inventory optimization, recommendation engines for retail. No mandatory obligations. This is where the vast majority of your portfolio lives, and where over-governance quietly destroys value.

The strategic insight: the Act is designed to concentrate scrutiny. If your governance program treats all 200 models with equal rigor, you have misread the entire regulatory intent. Proportionality is not a nice-to-have; it is the operating principle the legislators encoded.

Watch the foundation-model layer separately

There is a fifth track that cuts across the tiers: general-purpose AI (GPAI) models, including large language models. If you build on top of a foundation model, some obligations sit with the provider (OpenAI, Anthropic, Mistral). But if you fine-tune substantially, or if the model exceeds the systemic-risk compute threshold (10²⁵ FLOPs), obligations can flow to you. The practical CDO move: maintain a register of which foundation models you consume, whether you fine-tune them, and what documentation the provider gives you. When your legal team asks "are we a provider or a deployer of this GPAI system?", you need the answer in a spreadsheet, not a Slack archaeology dig.

Building the triage function: from 200 systems to a ranked queue

Here is the Monday-morning problem. You have an AI inventory (if you don't, that's the prerequisite lesson). Now you need to classify each entry into a tier without hiring a legal team to review every model. The answer is a two-stage funnelfunnelThe customer journey from awareness to purchase, typically Awareness, Interest, Consideration, Decision, Action, with prospects narrowing at each stage.Voir la définition complète →: a cheap automated screen, then a human adjudication for anything that trips a wire.

Stage one, the screening questionnaire. Attach five questions to every system in your registry, answered by the product owner, not the data scientist:

1. Does the system make or materially inform a decision about a specific individual?

2. Does that decision affect their access to employment, credit, education, healthcare, public benefits, or legal standing?

3. Is the system used by law enforcement, migration, or the justice system?

4. Does it use biometric data or infer emotional/protected characteristics?

5. Does it interact directly with a person or generate synthetic media?

Any "yes" to Q3 or Q4 escalates immediately. A "yes" to Q1 *and* Q2 flags high-risk. A "yes" only to Q5 flags limited-risk transparency obligations. All-no lands in minimal risk. You can encode this as a decision table so classification is reproducible and auditable rather than a matter of opinion:

# ai_risk_triage_rules.yaml — deterministic first-pass classifier
rules:
  - id: prohibited_check
    if: uses_social_scoring OR realtime_biometric_public OR workplace_emotion_recognition
    then: PROHIBITED           # halt: legal review before any further work
  - id: high_risk_domain
    if: affects_individual AND domain in [employment, credit, education,
         essential_services, critical_infra, law_enforcement, migration, justice]
    then: HIGH_RISK
  - id: transparency_only
    if: interacts_with_human OR generates_synthetic_content
    then: LIMITED_RISK
  - id: default
    then: MINIMAL_RISK
review:
  HIGH_RISK: mandatory_human_adjudication   # never auto-final on high-risk
  PROHIBITED: mandatory_human_adjudication

The point of the config isn't the syntax, it's the discipline. Classification becomes a policy artifact you can version, audit, and defend to a regulator, instead of a judgment that evaporates when the analyst who made it leaves.

Stage two, human adjudication. Never let the automated screen finalize a high-risk or prohibited label. The edge cases are where the money is. A credit model that only produces an internal risk score a human loan officer can override may or may not be high-risk depending on how much the human actually defers to it. This is the "meaningful human oversight" question, and it turns on operational reality, not org-chart theater. If your loan officers approve 99.3% of the model's recommendations, you do not have human oversight, you have a rubber stamp with a pulse, and a regulator will treat the system as high-risk.

What "high-risk" actually costs you operationally

When a system lands in the high-risk tier, the Act imposes a specific compliance stack. As a CDO, you need to know what each obligation demands of your teams:

• Risk management system: a continuous, documented process, not a one-time sign-off. Budget for ongoing headcount.
• Data governance: training, validation, and test datasets must be relevant, representative, and examined for bias. This is where your existing governance function pays dividends, but you now need documented evidence of the examination, not just the practice.
• Technical documentation and logging: the system must automatically log its operation to enable traceability. Design this in; retrofitting logging into a live model is brutal.
• Transparency to deployers: clear instructions for use.
• Human oversight: built-in, not bolted on.
• Accuracy, robustness, cybersecurity: documented performance thresholds and adversarial resilience.
• Conformity assessment before market placement, and registration in the EU database.

The critical distinction most CDOs get wrong: are you a provider or a deployer? If you develop the high-risk system (or substantially modify one), you carry the provider's full obligations. If you merely use someone else's high-risk system, you're a deployer with a lighter, but real, set: ensure human oversight, monitor operation, keep logs, and use it according to instructions. Most enterprises are deployers of vendor tools and providers of their in-house builds simultaneously. Your registry must tag which hat you wear for each system, because the obligations and the liability differ sharply.

Proportionality as a design principle, not an afterthought

The reason this lesson exists is that fear produces bad governance. When the fines are cited, up to €35M or 7% of global turnover for prohibited-system violations, leadership's instinct is to lock everything down. That instinct will make you slower than competitors on the 90% of your portfolio that is minimal-risk, while giving you false comfort that you've "done AI governance."

The mature posture is tiered controls that map to tiered risk. Concretely:

• Minimal-risk systems: register them, apply your standard data-quality and model-monitoring practices, and move on. No conformity assessment, no special documentation. Do not let a well-meaning risk committee invent obligations the Act doesn't require.
• Limited-risk systems: implement the transparency disclosure and audit that it actually fires. The failure mode here is a chatbot that never tells users it's a bot, or generated images without provenance labels. Cheap to fix, embarrassing to miss.
• High-risk systems: the full stack above, plus, and this is the CDO's genuine value-add, a *pre-deployment gate*. No high-risk system ships without conformity assessment sign-off. Wire this into your release pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → as a hard block, the same way you'd block a deploy that fails security scanning.

Consider how Klarna approached its customer-service AI. When they deployed a large-scale assistant, the system interacts directly with customers, a limited-risk transparency trigger, not high-risk, because it handles service inquiries rather than credit decisions. Their credit-underwriting models, by contrast, sit squarely in high-risk. Same company, same AI investment, two entirely different governance regimes. A CDO who governs both identically has either strangled the chatbot or under-protected the underwriting engine. Proportionality means you get this split right.

The timeline is your planning constraint

The Act phases in, and your roadmap must sequence to it:

• February 2025: prohibited-practice bans and AI-literacy obligations already live.
• August 2025: GPAI model obligations apply.
• August 2026: the bulk of high-risk obligations apply.
• August 2027: high-risk obligations for AI embedded in regulated products.

This staging is a gift. It tells you exactly where to spend first: confirm you have no prohibited systems (existential), then inventory your GPAI dependencies, then build the high-risk conformity stack ahead of the 2026 deadline. A CDO who front-loads the prohibited-systems audit and back-loads the high-risk documentation build is sequencing intelligently. One who tries to boil the ocean in Q1 is burning credibility.

Where the judgment really lives

The Act gives you tiers; it does not give you certainty at the boundaries. The genuine CDO judgment calls are:

• Borderline high-risk: an HR tool that "suggests" interview questions, does it "materially inform" an employment decision? When in doubt, treat it as high-risk provisionally and let legal argue it down. It is far cheaper to over-document and reclassify than to under-document and reconstruct history under regulatory pressure.
• Substantial modification: if you fine-tune a vendor model heavily, you may become the provider. Set a threshold with legal for what "substantial" means in your context, and log every modification against it.
• Extraterritorial reach: the Act applies if your system's output is used in the EU, even if you're headquartered elsewhere. A US-based CDO cannot treat this as someone else's problem.

Key Takeaways

1. Classify by use case, not by model. Build a deterministic triage screen tied to your AI registry that routes each system to a tier based on who it affects and what decision it drives, not on its technical sophistication.

2. Audit for prohibited systems first, this quarter. The bans are already in force and carry the largest fines. Everything else is a planning problem; this is an existential one.

3. Tag provider vs. deployer for every system. Your obligations and liability diverge sharply, and most enterprises are both simultaneously. Get this into the registry now.

4. Match control intensity to risk tier, and defend the minimal-risk 90% from over-governance. Proportionality is the Act's operating principle; violating it downward is illegal, violating it upward just makes you slow.

5. Wire a hard pre-deployment gate for high-risk systems into your release pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → before the August 2026 deadline, and design logging and documentation in from the start, retrofitting them into live models is the most expensive path.