# Building a model risk framework for safety-critical AI
In 2018, an Uber test vehicle in Tempe, Arizona struck and killed a pedestrian. The perception system detected her 5.6 seconds before impact but kept reclassifying her: vehicle, then unknown, then bicycle. The software was not designed to expect a person crossing outside a crosswalk. That is a model risk failure, and no financial provision covers it.
Banks have spent 25 years building discipline around models that misprice risk. Automotive AI needs the same discipline, but the loss function is a human life, not a bad loan. This lesson adapts banking-grade model risk management (MRM) to perception, prediction, and planning models in vehicles.
Model risk is the risk of loss (financial, reputational, or physical harm) from a model that is wrong or misused. The concept comes from the US Federal Reserve and OCC (Office of the Comptroller of the Currency) guidance SR 11-7, the foundational text on model risk management. Read the original: it is short and surprisingly readable (SR 11-7 guidance).
SR 11-7 says two things that transfer perfectly to cars:
1. A model is any quantitative method that turns input data into a decision. A pedestrian classifier qualifies.
2. Model risk comes from two sources: the model can be fundamentally wrong, and the model can be used incorrectly. Both apply to a lane-keeping system used on a road type it was never validated for.
The difference: a mispriced swap costs money. A missed pedestrian costs a life. So we keep the framework and raise the bar.
You are building this framework inside real law.
Name these bodies correctly when you talk to a regulator. Vague references signal you have not read the standards.
Banks tier models by materiality. You tier by harm potential. Build a simple tiering matrix combining severity (how bad if it fails) and controllability (can the driver or system recover).
| Tier | Description | Example |
|------|-------------|---------|
| Tier 1 | Failure can directly cause death, no human fallback in time | Pedestrian detection in an L3/L4 system at speed |
| Tier 2 | Failure contributes to harm, driver can intervene | Lane-departure warning, adaptive cruise |
| Tier 3 | Comfort or convenience, no safety path | Cabin gesture recognition, parking assist chime |
The tier sets the governance intensity. Tier 1 gets independent validation, extensive edge-case testing, and a formal sign-off. Tier 3 gets a lightweight review. Do not spend Tier 1 effort on Tier 3 models, or your safety team will drown and the real risks get missed.
The ODD (Operational Design Domain) is the specific conditions under which the model is validated to work: road types, weather, lighting, speed range, geography. This is your single most important control.
Most "AI failures" are actually ODD violations: a model used outside its validated envelope. The Tempe case involved a system operating at night against a scenario it was not built to handle.
Write the ODD as a contract. Example for a highway pilot:
Then enforce it in code. The vehicle must detect when it is leaving the ODD and hand back control safely.
SR 11-7's core principle: the team that builds a model cannot be the only team that judges it. You need effective challenge from a group with authority, competence, and independence.
For a Tier 1 perception model, independent validation means:
Here is a minimal validation gate expressed as code. The point is that acceptance is explicit and non-negotiable, not a vibe.
def perception_gate(metrics, odd_ok):
# Tier 1 acceptance thresholds (illustrative, set by safety team)
return (
metrics["pedestrian_recall"] >= 0.995 and
metrics["false_negative_rate_night"] <= 0.005 and
metrics["worst_subgroup_recall"] >= 0.99 and
odd_ok # model correctly detects ODD boundaries
)The thresholds are set by your chief safety officer, not chosen to make the model look good. Note the worst_subgroup_recall: an average that hides a weak subgroup is a lawsuit waiting to happen.
🎬 [VIDEO: "How Tesla's Autopilot and Full Self-Driving Actually Work" - youtube.com - clear breakdown of the perception-to-planning pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → in a real system]
A governance gate is a checkpoint where a model cannot advance to the next stage without documented sign-off. Your chief safety officer (CSO) is personally accountable, so the gate must give them defensible evidence.
Design three gates:
Gate A: Development to validation. Requires a completed safety case: a structured argument, backed by evidence, that the system is acceptably safe for its ODD. ISO/PAS 8800 expects this. Also requires a documented ODD and a data sheet describing training data provenancedata provenanceData lineage maps how data moves and transforms across systems, from origin to consumption, showing where it came from, what changed it, and where it goes.Voir la définition complète →.
Gate B: Validation to limited deployment. Requires independent validation passed, edge-case results, subgroup analysis, and a monitoring plan. Deployment is geofenced or shadow-mode first.
Gate C: Limited to full deployment. Requires field data from limited deployment showing real-world performance matches lab results, plus an incident and rollback procedure.
Each gate produces a signed artifact. When NHTSA or a court asks "how did you decide this was safe," you hand over the safety case. That is the difference between a defensible decision and a headline.
Vérification des acquis
1. According to SR 11-7 as adapted in this lesson, why does a pedestrian classifier qualify as a 'model' subject to model risk management?
2. The lesson describes a lane-keeping system used on a road type it was never validated for. Which SR 11-7 source of model risk does this primarily illustrate?
3. What is the core reason the lesson argues automotive AI should 'keep the framework and raise the bar' relative to banking MRM?
4. Select ALL correct answers about how the Uber Tempe incident illustrates model risk concepts.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers about the regulatory and standards backdrop described for safety-critical automotive AI.
Sélectionnez toutes les réponses correctes.
Banking learned the hard way that models decay. A credit model trained pre-2008 failed in the crash. Perception models decay too, through data drift: new vehicle designs, new e-scooter shapes, faded road markings, seasonal changes.
Build continuous monitoring:
A worked example. Suppose your fleet drives 2 million km per month and logs 40 safety-relevant disengagements. That is 1 per 50,000 km. If next month it rises to 80 disengagements over the same distance (1 per 25,000 km), the rate has doubled. That is a trigger to investigate, not to wait for a crash. Set the alarm threshold in advance and in writing, so nobody argues about it during an incident.
Every Tier 1 model needs a defined fallback: what the system does when it is uncertain or leaving its ODD. Options include a minimal risk maneuver (controlled slowdown and stop in a safe location) or a timed handover to the driver with escalating alerts.
The failure path is part of the model risk framework, not an afterthought. A perception model that fails gracefully into a safe stop is lower risk than a more accurate one that fails silently.