How bird's-eye-view mapping works, and where it breaks

I recently got curious to know how bird's-eye-view (BEV) mapping for perception in autonomous driving works: how six camera images around a vehicle become a top-down vectorized map of lane dividers, pedestrian crossings, and road boundaries. I studied this by running a pretrained MapTRv2 model on the nuScenes mini dataset, tracing tensor shapes through the full pipeline (lift-splat projection into a ground-plane grid, then a DETR-style decoder that refines 50 polylines over six layers), and then deliberately breaking it: twenty perturbations across calibration error, camera dropout, and image degradation, plus a temporal fusion experiment.

It was interesting to see how lopsided the failure modes are. A 10 cm error in camera height costs essentially nothing, while half a degree of rotation already hurts, because angular error projects onto the ground as range times its tangent. Dropping the rear camera is far worse than dropping the front one, simply because it has the widest field of view, so losing it empties the largest wedge of the map, the region behind the ego vehicle. But although it is the single worst per-frame failure, it is also the most forgivable one when the vehicle is moving forward: with temporal fusion, the road now behind the vehicle was seen by the forward cameras a moment ago, so memory recovers the loss almost entirely, while a front-camera failure cannot be recovered because the road ahead has never been seen. Beyond that, fusing predictions across frames never improved accuracy in any configuration I tried; it only trades accuracy for temporal stability. You can see the full write-ups and code in my bev_mapping repo.