How ADAS perception stacks turn sensors into driving decisions

# How ADAS perception stacks turn sensors into driving decisions

A car traveling at 70 mph covers about 103 feet every second. In the roughly 200 milliseconds it takes a human to react to a merging vehicle, that car has already moved 20 feet. An Advanced Driver Assistance System (ADAS, the suite of features like automatic emergency braking and lane centering) has to sense, decide, and act inside that same window, thousands of times per trip, without the luxury of a coffee break.

Let's trace one ordinary event: a car in the next lane drifts toward yours. How does the AI turn photons and radio waves into a steering nudge?

The scene: a lane-change event

You are driving on a highway. A sedan two car-lengths ahead in the right lane begins to merge left, into your path. Your car needs to notice, predict the sedan's trajectory, and either ease off the throttle or gently brake, all before you close the gap.

This single event flows through four stages: sense, fuse, predict, act. Each stage is a distinct AI or software problem.

Stage 1: Sense (the raw inputs)

Modern ADAS vehicles carry three complementary sensor types. None is good enough alone.

Camera. A camera sees color, text, and shape. It reads lane markings, brake lights, and the difference between a plastic bag and a rock. Its weakness: it struggles in glare, fog, and darkness, and it estimates distance poorly on its own.

Radar (radio detection and ranging). Radar bounces radio waves off objects and measures how they return. It directly measures distance and closing speed (via the Doppler effect, the frequency shift of a wave from a moving object) and works in rain and dark. Its weakness: low spatial resolution, so it can tell you "something metal is 40 meters ahead closing fast" but not whether it's a car or a road sign.

Lidar (light detection and ranging). Lidar fires laser pulses and times their return, building a precise 3D "point cloud" of the surroundings. It excels at exact shape and distance. Its weaknesses: cost and degraded performance in heavy fog or snow. Many mainstream ADAS systems skip lidar entirely and rely on camera plus radar; premium and robotaxi systems tend to include it.

In our lane-change scene:

• The camera sees a sedan and detects its left blinker and a slight lateral shift.
• The radar reports an object at 38 meters, closing at 4 mph, drifting laterally.
• The lidar (if present) confirms a vehicle-shaped point cluster changing position.

Three noisy, partial, sometimes conflicting stories. Now they have to be reconciled.

Stage 2: Fuse (combining sensors into one truth)

Sensor fusion is the process of merging multiple sensor streams into a single, more reliable model of the world. The core insight: each sensor's strength covers another's weakness.

There are two broad approaches.

Late fusion (object-level): each sensor first produces its own detections, then a fusion module reconciles them. Simpler, easier to debug, but throws away raw detail.

Early fusion (data-level): raw sensor data is combined before object detection, letting a neural network learn subtle cross-sensor patterns. More powerful, more compute-hungry, harder to validate.

A practical challenge is time and space alignment. The camera, radar, and lidar each capture at different rates and from slightly different physical mounting points. Fusion must timestamp every reading and transform them into one shared coordinate frame before they can be compared. A 30-millisecond mismatch at highway speed is a full meter of error.

Here is a simplified late-fusion association step in pseudocode:

# Match camera detections to radar tracks by predicted position
for cam_obj in camera_detections:
    best = None
    for radar_trk in radar_tracks:
        # distance between where each sensor thinks the object is
        gap = position_error(cam_obj.xy, radar_trk.xy)
        if gap < GATE_METERS and (best is None or gap < best.gap):
            best = radar_trk
    if best:
        fused = merge(cam_obj, best.velocity, best.range)
        world_model.update(fused)   # one object, richer than either sensor

The output of fusion is a clean world model: a list of tracked objects, each with a position, velocity, size, and class (car, pedestrian, barrier), plus a confidence score. Our drifting sedan is now a single tracked object: "vehicle, 38 m ahead-right, closing 4 mph, lateral velocity 0.5 m/s toward ego lane, confidence high."

("Ego" is the standard term for your own vehicle, the one the system is controlling.)

Stage 3: Predict (what happens next)

A snapshot isn't enough. The system needs to forecast the next few seconds.

Prediction models ask: given this object's recent motion, lane geometry, and blinker state, where will it be in 1, 2, and 3 seconds? Modern stacks use learned models trained on huge fleets of real driving data to estimate likely trajectories, often several at once with probabilities attached.

For our sedan: blinker on, steady lateral drift, gap closing. The model outputs a high probability that it will fully occupy the ego lane within about 2 seconds.

This is where AI earns its keep. A rules-only system ("if object crosses line, brake") would be jumpy and easily fooled. A learned predictor understands intent: a blinker plus lateral drift means "merging," while lateral drift alone near a curve might just mean "following the road."

Stage 4: Act (planning and control)

Now the system decides. Motion planning chooses a safe trajectory for the ego car; control translates that trajectory into steering, throttle, and brake commands.

Given the merging sedan, the planner weighs options:

• Maintain speed (unsafe, gap closes).
• Brake hard (safe but harsh, risks a rear-end from behind).
• Ease off throttle and apply light braking (smooth, opens the gap).

It picks the gentlest action that keeps a safe following distance, then hands a target deceleration to the control layer, which modulates the brakes. All of this repeats every few dozen milliseconds as fresh sensor data arrives.

The real-time constraint

Every stage runs on an automotive-grade compute platform under a strict time budget, often a target of well under 100 milliseconds end to end. This is a hard latency deadline (latency is the delay between input and response). Miss it and the car is acting on stale information. This is why ADAS software runs on dedicated chips, not a general laptop CPU, and why engineers obsess over shaving milliseconds.

Safety here is governed by frameworks like ISO 26262 (functional safety for road vehicles) and ISO 21448 / SOTIF (Safety Of The Intended Functionality, which addresses hazards from performance limits rather than outright failures, for example a camera blinded by sun glare). You can read an accessible overview of ADAS testing and ratings from Euro NCAP.

Why fusion beats any single sensor

Return to the drifting sedan and imagine each sensor working alone.

Camera only: in low sun glare, it might lose the sedan entirely for a few frames, or misjudge how fast the gap is closing.

Radar only: it knows something is closing but cannot confirm it is a car versus an overhead sign gantry, a classic cause of false braking events.

Lidar only: it sees the shape and distance perfectly but cannot read the blinker or brake lights, losing the intent signal.

Fused, the three cancel each other's blind spots. The camera supplies class and intent, radar supplies rock-solid closing speed even in glare, and lidar (when present) supplies exact geometry. Confidence rises, false alarms drop, and the planner can act decisively.

This is the core lesson for anyone evaluating automotive AI: performance is rarely about one heroic sensor or model. It comes from redundancy and disagreement handling. The hardest engineering questions are what to do when sensors conflict, how to degrade gracefully when one fails, and how to prove the whole chain is safe.

The business angle

For product and strategy leaders, this stack shapes real tradeoffs. Adding lidar raises cost per vehicle but can simplify validation and boost bad-weather performance. Camera-heavy approaches lower hardware cost but demand enormous training data and heavier compute. Neither is universally right, and the choice ripples into supplier contracts, pricing, and how a system earns safety ratings.

Key Takeaways

• ADAS perception is a pipeline: sense, fuse, predict, act, all repeating every few dozen milliseconds under a hard latency deadline.
• No single sensor is enough.