Why sensor fusion fails on real-world sites
At a proving ground, sensor fusion looks reliable: sensors are calibrated, streams are synchronized, the scene is clean.
On a real site, sensor desynchronization shows immediately — the camera outputs a frame, the lidar a point cloud, the radar a detection, and each stream arrives with its own latency.
Even at low speed, projections of the same obstacle diverge in space, and sensor data fusion begins to match measurements that refer to different points in time.
Hence the second type of failure — false positives. Glare and wet asphalt create blooming in optics, precipitation — noise in the point cloud, a roadside guardrail — re-reflection for radar.
Filtering rules tuned on dry daytime data let this noise through as an obstacle, and the vehicle brakes on an empty road. The reverse case is more dangerous: dense re-reflection masks a low object, and the obstacle does not enter any channel.
The third source — perception degradation when a channel fails.
Dusty optics, blinding by low sun, failure of one sensor: as long as consistency between streams is maintained, cross-checking pulls the system through, but with a silent channel dropout, the remaining sensors continue to output “confident” data without the overall picture.
The system does not understand that it has gone blind — this is the most expensive class of failures.
We break down how this is closed: a unified time scale for all streams, calibration under real operating conditions, cross-validation of measurements, and health monitoring of each channel while in motion.
What lidar, camera, and radar fusion provides
No single sensor provides the full picture: the camera goes blind in backlight, lidar loses points in heavy rain, radar is inferior in geometry.
We build lidar and camera fusion with radar so that the weakness of one channel is covered by the other two: we embed the sensor fusion layer into Autoware or into a proprietary control stack.
Object detection quality increases
A detection is confirmed only when at least two channels independently see it — by geometry, speed, and appearance.
Camera-based classification is verified by the rangefinder, so the false positive rate drops from whole percentages to tenths of a percent, and detection recall at long distances increases.
Robustness to interference instead of stopping
Rain, fog, dirt on optics, and oncoming light used to mean stopping and an operator dispatch. The system evaluates channel reliability and, when one degrades, reduces speed, increases distance, and drives the vehicle according to a fallback profile.
A whole class of perception failures is eliminated
Phantom objects from wet asphalt, double reflections, blind-spot dropouts — each such case previously required intervention. Spatiotemporal confirmation removes the class entirely, not individual episodes.
Fleet downtime is reduced
False braking accounted for the majority of remote engineer interventions. After channel fusion, the share of routes completed without operator involvement grows, and downtime per vehicle per shift falls.
Sensor fusion modes: when which approach is appropriate
Mode choice is determined not by model “power” but by which failures the system must survive and what resources are on board. Early fusion merges data before object detection, late fusion — after each channel has built its own world model. Between them lies a hybrid scheme, and on top of any of them radar redundancy is added.
| Option | When it is suitable | Limitations |
|---|---|---|
| Early fusion | Dense scene, many close objects, a unified environment picture is needed | Requires precise calibration and channel synchronization; sensitive to desynchronization and weather interference |
| Late fusion | Each channel already provides stable objects, simplicity of debugging and predictability are important | Details indistinguishable separately are lost; an error of one channel becomes fixed in the overall decision |
| Hybrid scheme | Conditions change: dense traffic in the city, sparse on the highway | More expensive to develop and maintain, harder to validate scenarios |
| Radar redundancy | Operation is needed when the camera fails or optics are blinded | Radar provides fewer features: object classes and their boundaries are determined approximately |
The practical selection rule is simple. Dense scene and overlapping fields of view — start with early fusion. Priority on robustness and failure interpretability — with late fusion.
Radar redundancy is added as a separate loop: it does not participate in the main decision, but prevents the system from going blind when one of the channels degrades.
Implementation stages: from sensor audit to handover
Work proceeds sequentially: each next stage relies on the results of the previous one, not on assumptions. This order removes the main risk of sensor fusion projects — when fusion is already assembled, but sensor desynchronization or blind spots are discovered only on the road.
- Sensor audit. We inventory the set: sensor types, relative placement, fields of view and their overlap, update rates, presence of factory and current calibration. We record which streams actually arrive synchronously and which will need to be brought to a common time.
- Fusion scheme. We choose the fusion level — raw data, already detected objects, or hybrid. The decision depends on priority: positioning accuracy versus robustness to loss of an individual channel.
- Synchronization and calibration. We bring streams to a unified time scale, compensate timestamp drift, and perform spatial alignment of the camera to the point cloud and radar. An error at this step produces false positives that cannot later be fixed by settings.
- Fusion tuning. We define channel priority rules, confidence thresholds, conflict resolution, and behavior under degradation — rain, fog, glare, optics contamination.
- Validation on runs. We run recorded scenarios and repeatable tests: detection recall, false positive rate, behavior on rare cases. Every change goes through regression on the same set so the result is reproducible.
- Solution handover. We hand over configurations, placement and alignment schemes, calibration regulations, retraining procedure, and rollback to the previous version. The customer’s team performs a run independently under our supervision.
After handover, the solution remains maintainable: it is clear what to change and by which rule when replacing a sensor or changing the layout.
Configurations, schemes, and regulations that remain with you
The unmanned system lives longer than the team that assembled it. Therefore we hand over not a “verbal solution” but a set of artifacts by which another team of engineers can deploy, recalibrate, and update the system without our involvement.
- System configurations — fusion parameters by each modality, confidence thresholds, weights when readings diverge, and behavior when one of the sensors drops out.
- Calibration data — lidar, camera, and radar alignment, time-shift corrections, and the recalibration procedure after reinstallation or unit replacement.
- Data flow diagrams — from raw measurements to the list of detected objects: where fusion occurs, how much is buffered, what is lost when a channel is delayed.
- Access policies — roles for engineers, analysts, and operations, rights to edit configurations, restrictions on exporting flight recordings, and an operations log.
- Validation sets — scenarios on real flight recordings and on synthetic data, metrics for missed and false detections.
- Maintenance regulations — model update procedure, rollback to the previous configuration, maintenance windows, and actions when quality degrades.
- Documentation — architecture description, deployment instructions, change log, and honestly recorded system limitations.
Handover is not an archive but a walkthrough: we go through each item with your engineers, show where the bottlenecks are and what changes when a sensor is replaced. Describe the task — we will come back with an estimate and a plan.
Case study: how false braking on a route was eliminated
Initial configuration: three channels — lidar, camera, radar — the sensor fusion layer merged them according to the rule “the most confident source wins.” In practice, this caused perception failures: reflections from wet asphalt and shop windows were marked as obstacles by lidar, the camera lost lane markings in backlight, and radar mistook the guardrail for an object in the lane.
The result was false braking on the route, up to 12 per one hundred kilometers, and constant perception cycle interruptions while driving.
What was changed
We abandoned choosing a single source: now each object receives a consistent position and speed estimate from all three sensors, and channel weight depends on conditions — rain, night, dust, oncoming light.
We added a plausibility check: an obstacle is confirmed only when two independent channels produce a matching trajectory for several consecutive cycles.
The computation was moved to NVIDIA DRIVE — the compute headroom is sufficient to maintain the full validation set without delay in the control loop.
What happened
False braking — 2 per one hundred kilometers instead of 12. Perception cycle interruptions per shift — a few instead of dozens.
The missed object rate at intersections dropped from 3% to 0.2%, and divergence between sensors no longer required manual readjustment after every ride in rain.
The implementation result rests not only on metrics: the unified environment model is described so that a new sensor or lidar replacement is added to the scheme within days — calibration and validation procedures are already worked out rather than reassembled for each unit of equipment.
What to do if sensors are already installed but data is not fused
Sensor diagnostics usually end the same way: each sensor individually is functional — lidar outputs a point cloud, camera frames, radar detections. The problem lies higher, in the data fusion layer: without it, the system sees three different worlds.
This leads to duplicate objects, track breaks during maneuvers, and “flickering” targets before braking.
| Symptom | Likely cause | What we check |
|---|---|---|
| One object is seen twice | Extrinsic parameter divergence | Residual error for each sensor pair |
| Track breaks in a turn | Timestamp delay | Unified time source and stream latency |
| Target “flickers” | Different update rate | Accumulation window and interpolation between frames |
| False positives in precipitation | Different modality robustness | Modality weights and confidence thresholds |
The improvement plan goes against intuition: measurements first, then edits. We bring timestamps to a single source and start capture on a common trigger so that the frame and point cloud describe the same moment.
Then we recalculate extrinsics using a target — in modern lidars, for example the Ouster series, the timestamp is set in hardware, and this removes some discrepancies even before calibration.
Next we choose the fusion scheme: early fusion at signal level is more accurate in geometry, late fusion — at object level — is cheaper in computation and more robust to loss of one modality.
Typical causes of failures after this are closed by tuning weights and thresholds, not by replacing sensors.
The outcome on already installed equipment: the missed target rate and number of ID switches drop by several times, and braking behavior becomes predictable.
Common technical doubts and answers
Synchronization, channel failure, sensor set expansion, and acceptance — these questions come up in almost every project, and it is better to answer them before the start rather than during testing. Below are short answers for each.
How are sensors synchronized with each other?
Streams are brought to one time scale in hardware — via a common reference signal, with the capture moment recorded. Otherwise, at speed, one target produces two different positions, and the tracker splits the object into two.
What happens if a sensor fails while driving?
The system enters a reduced mode: the target is confirmed by the remaining modalities, track weight decreases, but the loop does not break.
Diagnostics flags the lost stream, and onboard logic transitions the vehicle to a safe scenario — speed reduction or stopping in a permitted zone.
Can a new sensor be added later?
Yes, if the fusion layer is separated from the drivers: the source is connected with its own calibration and weights, and the core is not rewritten. There is one condition — calibration and timestamps are brought to a common format at the input.
How do you assess fusion quality?
We calculate recall and precision by class, false positive rate, robustness to dropouts, and behavior under poor visibility. We run regression on scenes cut from training — otherwise the metric shows memorization rather than performance.
Where is the boundary of contractor responsibility?
We cover the data pipeline, calibration, fusion, and model validation; the customer covers the onboard platform, access to sensors, and scenario acceptance. We fix the boundaries in writing before the start so that disputed areas do not surface during testing.
Let’s discuss the task and come back with an estimate and plan
To come back with an estimate and plan, rather than general considerations, we need baseline data about your task.
Tell us which sensors are already on board and how they work together — lidar, camera, and radar cover different parts of the picture, and their combination determines which discrepancies will have to be compensated algorithmically and which are solved at the mechanical level.
It is enough to describe the task in free form; we will clarify the rest ourselves:
- sensor composition: models, fields of view, relative placement, and rigidity of mounting relative to each other;
- operating conditions: illumination, precipitation, dust, vibration, temperature and altitude range;
- behavioral requirements: what the onboard system does when readings diverge and which class of errors is unacceptable;
- compute loop: what you currently have and constraints on mass, power, and heat;
- baseline data: whether there are flight recordings and annotations, and what can be used to verify data fusion quality;
- desired result: where the solution is applied and by which measurements you will accept it.
Based on these inputs, we will come back with an estimate and plan: work stages, intermediate validation points, and what acceptance will look like. Describe the task — we will discuss it substantively and say what is realistically achievable on your configuration.
