How Self-Driving Cars Work: Sensing, Planning and the Levels of Autonomy

Public discussion around self-driving cars often treats autonomy as a binary milestone—either a car can drive itself or it cannot. In automotive engineering, however, automated driving is a spectrum of distinct hardware capabilities, operational domains, and human supervision requirements. Understanding where autonomous vehicles stand today requires looking under the hood at four interconnected subsystems—perception, localization, prediction and planning, and actuation control—as well as the industry-standard taxonomy defined by SAE International. This guide breaks down how self-driving stacks operate and why handling rare roadway edge cases remains one of the hardest challenges in robotics.

The SAE J3016 Levels of Driving Automation (Levels 0 to 5)

To prevent confusion between driver-assistance features and true driverless vehicles, SAE International maintains standard J3016, which classifies systems into six levels based on who is responsible for monitoring the road and what happens when the system encounters a situation it cannot handle:

• Level 0 (No Driving Automation): The human driver performs all steering, braking, and acceleration continuously, though the car may provide momentary emergency warnings or automatic emergency braking.
• Level 1 (Driver Assistance): The vehicle controls either steering (lane centering) or speed (adaptive cruise control) at a given time, while the human performs the other task and monitors the environment constantly.
• Level 2 (Partial Driving Automation): The system controls both steering and acceleration/deceleration simultaneously under specific conditions, but the human driver must remain fully attentive, keep eyes on the road, and take over immediately at any moment. Most consumer highway-assist systems sold on passenger cars today are Level 2.
• Level 3 (Conditional Driving Automation): Within a strictly defined Operational Design Domain (ODD)—such as a divided highway in clear weather below a certain speed—the vehicle manages the entire driving task and the human is not required to monitor the road continuously, provided the human remains ready to resume control within a short notice window when the system requests a takeover.
• Level 4 (High Driving Automation): The vehicle performs all driving tasks and monitors the environment without any human intervention inside its approved Operational Design Domain (for example, a geofenced urban robotaxi zone or an industrial yard). If a failure or severe weather event occurs, the car must achieve a "minimal risk condition"—such as pulling safely to the shoulder—on its own, without relying on a human driver.
• Level 5 (Full Driving Automation): The system can drive everywhere a qualified human driver can, under all road, geography, and weather conditions, with no Operational Design Domain restrictions.

Subsystem 1: Multi-Modal Perception (Cameras, Radar, and Lidar)

Before an autonomous vehicle can decide where to steer, it must build a real-time 3D representation of the surrounding world. Most Level 4 architectures rely on sensor fusion—combining three complementary physical sensing modalities so that the weakness of one sensor is covered by the strengths of the others:

Cameras (Passive Optical Vision)

High-dynamic-range cameras surrounding the vehicle provide dense color, texture, and semantic detail. Cameras are essential for reading road signs, interpreting traffic light colors, recognizing brake lights or turn signals on other vehicles, and distinguishing lane markings. However, passive cameras do not measure distance directly and can be degraded by direct sun glare, heavy rain, fog, or darkness.

Radar (Radio Detection and Ranging)

Millimeter-wave radar transmits radio pulses that bounce off objects, directly measuring both range and relative radial velocity (via the Doppler effect) in a single cycle. Radar penetrates fog, rain, snow, and dust reliably and can track fast-approaching vehicles hundreds of meters away, though it has lower angular resolution than optics or lasers when discerning the exact shape of stationary objects.

Lidar (Light Detection and Ranging)

Spinning or solid-state lidar units emit millions of eye-safe near-infrared laser pulses per second and measure the time of flight of the returning photons. This produces a precise,three-dimensional point cloud of the environment with centimeter-level depth accuracy, day or night, making it straightforward to measure the exact geometry of curbs, pedestrians, cyclists, and overhanging obstacles.

Subsystem 2: Localization and Mapping

Standard consumer GPS is only accurate to within several meters—and in dense urban canyons surrounded by glass skyscrapers, satellite signals bounce off buildings, creating multipath errors large enough to place a car in the wrong lane. Autonomous vehicles achieve centimeter-level localization by combining three inputs:

1. Inertial Measurement Units (IMUs) and Wheel Odometry: High-frequency accelerometers and gyroscopes track the car's instantaneous acceleration and rotation hundreds of times per second, bridging gaps between external sensor updates.
2. Real-Time Kinematic (RTK) GNSS: Corrected satellite positioning provides a coarse global coordinate.
3. Sensor-to-Map Registration: In mapped operating zones, the vehicle matches the live 3D geometry captured by its lidar and cameras against a pre-surveyed high-definition (HD) map containing exact lane topology, crosswalk boundaries, and traffic light positions.

Subsystem 3: Behavior Prediction and Motion Planning

Once the vehicle knows where it is and what objects surround it, the software stack must look several seconds into the future:

• Intent and Trajectory Prediction: Neural networks analyze the posture and velocity history of every nearby pedestrian, cyclist, and vehicle to generate probability distributions over their likely paths over the next five to ten seconds (for example, predicting whether a pedestrian walking near a curb intends to step into the crosswalk or continue along the sidewalk).
• Behavioral and Trajectory Planning: The planner searches through thousands of candidate trajectories for the ego vehicle, scoring each path against a hierarchy of constraints: hard safety rules (never collide with an obstacle), traffic laws, contingency margins if another road user behaves unpredictably, passenger ride comfort (limiting sudden jerk and lateral acceleration), and progress toward the destination.
• Control Loop Execution: Once an optimal trajectory is chosen for the next fraction of a second, low-level PID or Model Predictive Control (MPC) loops translate that curve into exact steering-angle torque, throttle, and brake-pressure commands.

Modular Pipelines vs. End-to-End Neural Architectures

Historically, self-driving software stacks were built as strictly separate modules—perception output bounding boxes, prediction output trajectories, and a hand-engineered planner wrote the motion curve. While modular designs are easier to inspect and debug, information can be lost at the boundaries between modules (for instance, a bounding box around a traffic officer does not capture the subtle hand gesture the officer is making).

In recent years, architectures have increasingly adopted end-to-end learning or hybrid foundation models that process raw multi-camera and sensor streams jointly with planning tokens, while keeping a deterministic safety-envelope monitor ("safety shield") around the final actuator commands to guarantee physical collision bounds are never violated.

Why the "Long Tail" of Edge Cases Makes Autonomy Hard

Driving ninety-nine percent of normal highway or suburban miles is relatively tractable for modern robotics. The difficulty lies in the long tail of rare, novel situations that occur once every hundred thousand or million miles: a construction worker holding a reversible STOP/SLOW paddle while waving vehicles into an oncoming lane, a mattress falling off a truck ahead at highway speed, emergency vehicles blocking an intersection with contradictory cones, or snow obscuring every lane marking. Deploying Level 4 fleets safely requires not just road testing, but massive closed-loop simulation engines capable of replaying and mutating millions of synthetic edge-case scenarios before software updates reach public streets.

Frequently Asked Questions

Why is jumping from Level 2 to Level 3 or Level 4 so legally and technically difficult?

At Level 2, the human driver is legally and operationally the safety fallback at every millisecond; if the software gets confused, the human is already watching the road. At Level 4, there is no human safety net—the vehicle must have redundant braking, redundant steering power, redundant compute boards, and software capable of handling sensor failures and pulling over safely on its own.

Can a self-driving car work without an internet connection?

Yes. All safety-critical perception, prediction, and motion-control loops run locally on the vehicle's onboard compute cluster in real time, because cellular networks can experience latency spikes or dead zones. Cellular connectivity is used only for non-real-time tasks such as dispatching rides, receiving traffic routing updates, or requesting high-level guidance from a remote fleet operator when the car is stationary and blocked by an unusual road closure.

What does "Operational Design Domain" (ODD) mean?

An Operational Design Domain is the specific set of conditions under which an automated driving system is designed and validated to function safely—including geographic boundaries, road types, speed ranges, time of day, and weather thresholds. A Level 4 vehicle operates autonomously only inside its validated ODD.

Further Reading

• SAE International J3016: Taxonomy and Definitions for Terms Related to Driving Automation Systems — The global engineering standard defining Levels 0 through 5.
• NIST AI Risk Management Framework — Safety and assurance principles for physical and cyber-physical AI systems.