Clearing the safety and validation bar for AI you ship

# Clearing the safety and validation bar for AI you ship

A driver is asleep. The car is doing 70 mph on a highway at dusk. A truck ahead loses a mattress, which tumbles into the lane. The AI driving function has roughly one second to perceive the object, classify it, predict its motion, plan a maneuver, and execute a brake or lane change. If it gets this right, nobody notices. If it gets it wrong, a company faces a fatality, a recall, and a courtroom.

The hard question for any automotive team shipping AI is not "does it work in the demo?" It is "can we prove it is safe enough to sell, register, and defend?" That proof has a name in this industry: homologation (the formal approval that lets you legally sell a vehicle in a market) plus a defensible safety case.

This lesson walks through the three pillars teams use to clear that bar.

Pillar 1: ISO 26262 and functional safety

ISO 26262 is the international standard for functional safety in road vehicles. "Functional safety" means the system behaves safely even when a component fails. Think of a sensor that dies, a chip that flips a bit, or software that crashes.

The core tool is ASIL, the Automotive Safety Integrity Level. Every hazard gets rated from A (lowest) to D (highest), based on three factors:

• Severity: how bad is the harm?
• Exposure: how often is the vehicle in this situation?
• Controllability: can a driver or the system avoid the harm?

A brake-by-wire failure at highway speed is ASIL D. A stuck interior light is quality-managed, not safety-rated at all.

Here is the friction with AI. ISO 26262 was written for systems you can specify completely. You can write a requirement for an anti-lock braking algorithm and test every branch. A neural network trained on millions of images has no such tidy specification. You cannot enumerate everything it learned.

So teams do not certify the network itself under classic 26262. They wrap it in architecture they *can* certify:

• A redundant, rule-based safety monitor that overrides the AI if it commands something dangerous.
• A simpler, verifiable "safety envelope" that guarantees the car never exceeds physical limits, regardless of what the AI wants.

The AI proposes. The certified layer disposes.

Pillar 2: SOTIF, for when nothing broke but harm still happened

ISO 26262 covers failures. But AI has a scarier failure mode: everything works exactly as designed, and it still does the wrong thing.

That is the domain of SOTIF (Safety Of The Intended Functionality), standardized as ISO 21448. SOTIF exists because a perception system with zero hardware faults can still misread a white truck against a bright sky, or fail to recognize a pedestrian in an unusual pose.

SOTIF divides the world into four boxes:

1. Known safe scenarios (what you designed for).

2. Known unsafe scenarios you have found and fixed.

3. Unknown unsafe scenarios: the ones that hurt you.

4. Unknown safe scenarios (harmless surprises).

The entire engineering job is shrinking box 3. You do that by mining edge cases, running structured scenario analysis, and pushing unknown-unsafe cases into the known-and-mitigated column.

The UNECE World Forum for Harmonization of Vehicle Regulations publishes the regulatory framework many markets adopt, and it increasingly references both standards for automated driving approvals.

Why edge cases dominate the budget

The uncomfortable truth: the last 1 percent of scenarios consumes most of the validation effort. Normal highway driving is easy. What breaks systems is the long tail: a horse on the road, a flooded underpass, a construction worker waving contradictory hand signals, emergency vehicles with unusual light patterns.

You cannot design these away in advance. You have to find them, then prove your system handles them.

Pillar 3: billions of simulated miles

How do you find rare events? You cannot drive them into existence economically. A widely cited RAND analysis argued that proving an autonomous system is safer than humans by real-world driving alone could require hundreds of millions or even billions of miles, which would take fleets decades. Treat those specific figures as estimates, but the direction is not controversial: real miles are too slow and too expensive for the rare events that matter most.

So the workload moves to simulation.

Scenario-based simulation. Teams build a library of parameterized scenarios (a cut-in, a jaywalker, a stalled vehicle) and sweep every variable: speed, distance, lighting, weather, road friction. One scenario becomes thousands of tests.

Resimulation (log replay). Every mile a real fleet drives becomes an asset. You record the sensor data, then replay it against new software versions to check for regressions. Did the update that fixed the highway merge break the parking-lot behavior?

Synthetic edge-case generation. Where real data is thin, you generate it: rare weather, rare object types, adversarial placements a human tester would never think of.

The output is not "we drove a lot." It is a coverage argument: we have systematically explored the operational design domain and shown the function stays safe across it.

The operational design domain (ODD) is the precise envelope where the system is approved to operate: which roads, speeds, weather, and times of day. A system rated only for divided highways in clear weather below 60 mph has a narrow ODD, and that is often a feature, not a weakness. A tight ODD is easier to validate and easier to defend.

🎬 [VIDEO: "How Waymo Uses Simulation to Test Self-Driving Cars" — youtube.com — a look inside large-scale driving simulation and scenario testing]

Tying it together: the safety case

All three pillars feed one deliverable: the safety case. This is a structured, auditable argument that the system is acceptably safe, supported by evidence. Regulators want it. Your own legal team wants it. Insurers want it.

A common way to structure it is a claim, argument, evidence tree. In pseudocode form it looks like this:

CLAIM: The lane-keeping function is acceptably safe within its ODD.
  ARGUMENT: All identified hazards are mitigated to target ASIL.
    EVIDENCE: HARA (Hazard Analysis and Risk Assessment) document
    EVIDENCE: ASIL-D safety monitor verification results
  ARGUMENT: Residual SOTIF risk is acceptably low.
    EVIDENCE: 4.2M scenario-simulation runs, pass rate + failure triage
    EVIDENCE: field monitoring plan for unknown-unsafe cases
  ARGUMENT: The ODD is enforced at runtime.
    EVIDENCE: geofence + weather-degradation handover logic tests

Notice what this buys you in a liability case. If something goes wrong, the question is rarely "was the system perfect?" No system is. The question is "did the company act responsibly given the state of the art?" A traceable safety case, showing you identified the hazard, engineered a mitigation, and validated it, is your strongest evidence that you did.

What this means for how you build

The validation bar reshapes the whole product organization, not just the safety team.

Data pipelines become safety infrastructure. Every fleet mile that can be replayed is a validation asset. Teams that log richly and retrievably validate faster.

Field monitoring never ends. SOTIF is a lifecycle, not a launch gate. You ship, then you watch for the unknown-unsafe cases that only appear at scale, then you fix and re-validate. Over-the-air updates make this loop tighter, but every update reopens the validation question.

Narrow beats broad. A tightly scoped ODD you can fully defend often ships years before an ambitious one you cannot. Many production driver-assistance features succeed precisely because they refuse to operate outside their envelope.

The safety case is a business document. It gates market access, shapes insurance terms, and caps liability exposure. Executives who treat it as paperwork are mispricing risk.

Key Takeaways

• ISO 26262 handles failures; SOTIF handles correct-but-wrong behavior. AI needs both, because a fault-free neural network can still misperceive the world.
• You cannot certify a neural network directly, so you wrap it. A verifiable safety monitor and a physics-based safety envelope let you make guarantees the AI itself cannot provide.
• Simulation is the only economical way to reach the rare events that matter. Scenario sweeps, log replay, and synthetic edge cases build a coverage argument, not just a mileage total.
• A narrow, enforced ODD is a competitive advantage. Defining exactly where the system works makes it faster to validate, approve, and defend.