How ADAS perception stacks turn sensors into driving decisions

Lidar (light detection and ranging). Lidar fires laser pulses and times their return, building a precise 3D "point cloud" of the surroundings. It excels at exact shape and distance. Its weaknesses: cost and degraded performance in heavy fog or snow. Many mainstream ADAS systems skip lidar entirely and rely on camera plus radar; premium and robotaxi systems tend to include it.

In our lane-change scene:

• The camera sees a sedan and detects its left blinker and a slight lateral shift.
• The radar reports an object at 38 meters, closing at 4 mph, drifting laterally.
• The lidar (if present) confirms a vehicle-shaped point cluster changing position.

Three noisy, partial, sometimes conflicting stories. Now they have to be reconciled.

Stage 2: Fuse (combining sensors into one truth)

Sensor fusion is the process of merging multiple sensor streams into a single, more reliable model of the world. The core insight: each sensor's strength covers another's weakness.

There are two broad approaches.

Late fusion (object-level): each sensor first produces its own detections, then a fusion module reconciles them. Simpler, easier to debug, but throws away raw detail.

Early fusion (data-level): raw sensor data is combined before object detection, letting a neural network learn subtle cross-sensor patterns. More powerful, more compute-hungry, harder to validate.

A practical challenge is time and space alignment. The camera, radar, and lidar each capture at different rates and from slightly different physical mounting points. Fusion must timestamp every reading and transform them into one shared coordinate frame before they can be compared. A 30-millisecond mismatch at highway speed is a full meter of error.

Here is a simplified late-fusion association step in pseudocode:

The output of fusion is a clean world model: a list of tracked objects, each with a position, velocity, size, and class (car, pedestrian, barrier), plus a confidence score. Our drifting sedan is now a single tracked object: "vehicle, 38 m ahead-right, closing 4 mph, lateral velocity 0.5 m/s toward ego lane, confidence high."

("Ego" is the standard term for your own vehicle, the one the system is controlling.)

Stage 3: Predict (what happens next)

A snapshot isn't enough. The system needs to forecast the next few seconds.

Prediction models ask: given this object's recent motion, lane geometry, and blinker state, where will it be in 1, 2, and 3 seconds? Modern stacks use learned models trained on huge fleets of real driving data to estimate likely trajectories, often several at once with probabilities attached.

For our sedan: blinker on, steady lateral drift, gap closing. The model outputs a high probability that it will fully occupy the ego lane within about 2 seconds.

This is where AI earns its keep. A rules-only system ("if object crosses line, brake") would be jumpy and easily fooled. A learned predictor understands intent: a blinker plus lateral drift means "merging," while lateral drift alone near a curve might just mean "following the road."

Stage 4: Act (planning and control)

Now the system decides. Motion planning chooses a safe trajectory for the ego car; control translates that trajectory into steering, throttle, and brake commands.

Given the merging sedan, the planner weighs options:

• Maintain speed (unsafe, gap closes).
• Brake hard (safe but harsh, risks a rear-end from behind).
• Ease off throttle and apply light braking (smooth, opens the gap).

It picks the gentlest action that keeps a safe following distance, then hands a target deceleration to the control layer, which modulates the brakes. All of this repeats every few dozen milliseconds as fresh sensor data arrives.

The real-time constraint

Every stage runs on an automotive-grade compute platform under a strict time budget, often a target of well under 100 milliseconds end to end. This is a hard latency deadline (latency is the delay between input and response). Miss it and the car is acting on stale information. This is why ADAS software runs on dedicated chips, not a general laptop CPU, and why engineers obsess over shaving milliseconds.

Safety here is governed by frameworks like ISO 26262 (functional safety for road vehicles) and ISO 21448 / SOTIF (Safety Of The Intended Functionality, which addresses hazards from performance limits rather than outright failures, for example a camera blinded by sun glare). You can read an accessible overview of ADAS testing and ratings from Euro NCAP.

Why fusion beats any single sensor

Return to the drifting sedan and imagine each sensor working alone.

Camera only: in low sun glare, it might lose the sedan entirely for a few frames, or misjudge how fast the gap is closing.

Radar only: it knows something is closing but cannot confirm it is a car versus an overhead sign gantry, a classic cause of false braking events.

Lidar only: it sees the shape and distance perfectly but cannot read the blinker or brake lights, losing the intent signal.

Fused, the three cancel each other's blind spots. The camera supplies class and intent, radar supplies rock-solid closing speed even in glare, and lidar (when present) supplies exact geometry. Confidence rises, false alarms drop, and the planner can act decisively.

This is the core lesson for anyone evaluating automotive AI: performance is rarely about one heroic sensor or model. It comes from redundancy and disagreement handling. The hardest engineering questions are what to do when sensors conflict, how to degrade gracefully when one fails, and how to prove the whole chain is safe.

The business angle

For product and strategy leaders, this stack shapes real tradeoffs. Adding lidar raises cost per vehicle but can simplify validation and boost bad-weather performance. Camera-heavy approaches lower hardware cost but demand enormous training data and heavier compute. Neither is universally right, and the choice ripples into supplier contracts, pricing, and how a system earns safety ratings.

Key Takeaways

• ADAS perception is a pipeline: sense, fuse, predict, act, all repeating every few dozen milliseconds under a hard latency deadline.
• No single sensor is enough. Cameras read intent and class, radar measures closing speed in any weather, lidar delivers precise 3D geometry. Fusion combines their strengths.
• Prediction, not just detection, is where AI adds value. Forecasting a merging car's intent enables smooth action instead of jumpy, rule-based reactions.
• Safety is a systems problem. Standards like ISO 26262 and SOTIF address both outright failures and performance limits such as sensor blinding.
• Sensor choice is a business decision. Camera-heavy versus lidar-equipped stacks trade hardware cost against data needs, compute, and validation effort.