Why does every self-driving car company use different sensors and different mounting positions? Although there is still a large variety in sensor sets, there are some recurring patterns to be found, like high resolution lidars mounted on the roof center of autonomous vehicles. This article aims to provide an overview of these patterns and to explain the reasoning behind the different approaches.

🛈 Disclaimer / scope

  • This article focusses on sensor positioning and the impact of the field of view (FOV) of sensors on the scenarios autonomous cars can manage. Details about sensors and sensor technology are not a major subject of this article.
  • This article focusses a more on camera and lidar than on radar. Other sensors like ultrasonic or thermal cameras are highly interesting for autonomous driving as well but are not a major subject of this article.
  • For the sake of simplicity, sensor coverage is visualized as continuous volumes.
  • The FOV of the model sensors used for the considerations below reflects “standard sensors” used in autonomous driving, see Lidar comparison, Radar comparison.

Abbreviations

FOV:               Field Of View
ADAS:            Advanced Driver Assistance System
AV:                 Autonomous Vehicle

General Considerations

Given their individual features, the sensors of an autonomous car determine its operational design domain (ODD) – the environments in which the car can be deployed safely. Identifying the sensors required to “solve” an ODD is complex since there is a plethora of common cases and edge cases to consider, while all sensors have individual traits and limitations – each sensor is chosen as a trade-off between accuracy, range, sampling rate, FOV, cost and overall system complexity.

Scenarios

A common way to identify the sensor set needed to solve an ODD is to capture a variety of scenarios, that are fed into simulation to check under which circumstances a sensor set would miss objects in the environment. The performance of different sensor sets is compared, and promising configurations are improved iteratively. This data-driven approach allows to use Machine Learning in order to identify the optimal configuration. Scenarios can be collected automatically from cars in the field or can be designed manually. Below is a selection of challenging scenarios and edge cases:

Mid-air and ceiling obstacles

Obstacles blocking a vehicle’s path that are not connected to the ground, e.g. open truck doors, protruding truck freight, tree branches, boom gates or low bridges, are missed by sparse or low lidars, are very challenging for camera and can even be hard for radars to associate correctly.

If a lidar sensor with small vertical FOV is mounted too low, none of the lidar beams it emits might hit a protruding obstacle.

Protruding obstacle are not detected if the vertical resolution of the lidar sensor is too low, and no beam hits the obstacle.

Cyclist passing by

When taking a turn to the right, special caution must be paid for traffic participants that could be cut off by the turn. While big vehicles are comparably easy to detect (via Radar, e.g. for lane change assist), light traffic participants such as cyclists can be significantly harder to spot, sensors mounted on the roof center.

A cyclist riding next to the vehicle is not in the FOV of lidar sensor mounted on the roof center.

Crossing with occlusion

To reduce the risk of collisions to a minimum when entering an occluded crossing, it is crucial to be able to observe cross-traffic lanes as early as possible, e.g. by sensors mounted at the front of the vehicle.

A centrally mounted sensor would not be able to perceive traffic coming from an occluded crossing lane.

A sensor facing sideways close to the vehicle front can peek into the crossing lane and detect the approaching cross-traffic.

Low obstacle besides car

Close, low obstacles, such as curbstones, guardrails, bushes etc. are a challenge when using sensors with small vertical FOV because they can either be overlooked or they can shadow traffic participants behind them.

A low obstacle is overlooked by the sensor because it is too short.

A close, low obstacle is completely occluding traffic behind it.

Slope

Slope has a major impact on the requirements towards the vertical FOV of sensors. When approaching ramps and underpasses, the coverage on other traffic participants is reduced. In addition, the geometry of the road surface must be compensated for object detection; otherwise ramps will be classified as obstacles, and actual obstacles out of the FOV will be missed. Particular challenging are intersections with ascending or descending crossing lanes.

A ramp in front of the vehicle can be erroneously interpreted as an obstacle, while the second vehicle is not covered by the sensor at all.

Sensor Colocation

When sensors are mounted apart from each other, they’ll have a different perspective on the vehicle’s environment (parallax) which may lead to a situation in which one sensor perceives an object, while it is occluded for the other sensor. This is a challenge for sensor data fusion: for high level fusion (fusion of objects), there’ll be contradicting detections from the sensors. For low level fusion (fusion of raw sensor data), the parallax will result in areas that can’t be matched between the sensors and consequently ‘holes’ in the fused sensor data. These effects are particularly strong for the close environment.

To reduce these issues to a minimum, different sensor modalities, particularly camera and lidar, are often colocated: the sensors are mounted as close as possible to each other.

Coverage and redundancy

Depending on the intended degree of automation and the target ODD, the requirements towards the sensor set vary substantially.

For a highway lane keeping feature with the driver supervising, a front facing radar and camera can be sufficient.

For autonomous driving in urban environments, 360° horizontal coverage across all three major sensor modalities (camera, lidar and radar) is required in order to reliably detect traffic participants in the relevant scenarios while leveraging the different modalities to compensate their respective weaknesses. In fact, 360° are necessary, but won’t suffice; a 360° spinning lidar on top of the vehicle is insufficient to detect obstacles in many cases (see lidar section below) and needs to be complemented by further lidars.

Commercial considerations

Having solved an ODD enables offering a product. Whether there’ll be a positive business case for this product or not will depend on many further factors, such as the nature of the service provided by the AV, region where the service is being provided etc.

This article won’t explore commercial considerations, but it is vital for sensor set design to acknowledge that, apart from sensors’ contribution to solving the ODD, cost, reliability and production scalability are decisive factors for the deployment of large fleets.

Sensor Modalities

Lidar

Lidars are a comparably new technology in the automotive industry; they used to be more common in robotics. With the advance of ADAS and autonomous driving, this is about to change. While first series cars use low resolution Lidars already, high resolution Lidars allowing detection at range and even classification are currently reserved for self-driving cars with generous sensor budgets.

There are various different lidar sensors on the market, different in wavelength, resolution, and technology – to get familiar with the technology behind the sensors, read more here.

Single lidar on roof center

Variations

  • Sensor can be elevated to reduce blind spot by occluding roof.
  • Sensor can be moved from the roof center to the roof front edge, which improves front perception while decreasing rear perception.

Pros

  • Simple setup, no effort to synchronize and align multiple point clouds.
  • 360° coverage with one sensor.
  • Good overview, ability to look over other traffic participants.

Cons

  • Blind spots for low objects in all directions, particularly to the rear.
  • Sensor must be elevated from roof to leverage full vertical FOV, the high position causes mechanical challenges in case of abrupt deceleration and makes it impossible to enter areas with low ceiling.

Multiple Lidars on the roof

Variations

  • Sensors can be tilted to decrease blind spots.
  • Sensors can be mounted on the edges of the vehicle roof or complemented by additional sensors on the roof.

Pros

  • No occlusion by vehicle roof
  • Tilted sensors reach optimal coverage among roof mounted configurations.

Cons

  • Higher complexity in terms of integration and point cloud fusion compared to single sensor
  • Tilting of the sensors is limited by their vertical FOV. Too few tilting results in blind spots near the vehicle while too much tilting limits coverage on elevated objects.

Front lidar

Variations

  • Lidars with bigger vertical FOV enable steeper slopes.
  • Higher mounting points behind windshield or at the front roof edge center allow to look over other traffic participants and low obstacles at the cost of missing low obstacles.

Pros

  • Simple setup, good integration into vehicle,
  • On flat surfaces: Lidar beams are almost parallel to the road surface; hence no road surface reflections are returned. This allows to interpret the returns directly as a distance to obstacles.
  • Good for detection of vehicles in the front on highways (ACC).

Cons

  • On slopes; sensor will return ground plane as an obstacle for ascending roads and no returns for descending roads,
  • Insufficient overall coverage for urban AD if not combined with other sensors.

Side view lidars

Variations:

  • Higher horizontal FOV and vertical FOV lidars enable full coverage of the vehicle’s sides.
  • Different positions: position behind front wheel case may simplify integration, position below side mirror allows to look over low obstacles.

Pros:

  • Enables Crossing with occlusion scenario: The vehicle can detect cross-traffic at intersections without entering and potentially blocking the traffic.

Cons:

  • Sensor with low vertical FOV will have issues with slopes

Protruding 360° lidars on the vehicles’ sides provide 180° coverage.

Camera

Cameras (as opposed to Lidars, radars and ultrasonic sensors) are passive sensors, that collect light reflected from the environment. While cameras offer a comparably high resolution, their data is subject to a variety of environment influences (darkness, rain, fog, low sun, spray, snow, dust, …). Camera data needs to be interpreted by computationally expensive computer vision algorithms or Deep Learning in order to provide useful information for ADAS and autonomous driving, while the data from distance measuring sensors such as Lidar can used (almost) as-is for collision avoidance. On the positive side camera data (once interpreted) provides the semantically richest representation of the vehicle’s environment.

Wide angle (“Fisheye”) cameras have a large FOV, but their images are subject to lens distortion that needs to be corrected before further processing. Lens distortion correction removes parts of the image and consequently reduces the FOV. In addition, the areas close to the edges of the image may be blurred and less reliable for image recognition tasks.

360° coverage by roof-mounted wide-angle cameras

Many series cars are equipped with a set of wide-angle cameras to provide a bird’s-eye view of the close environment that makes parking easier. Usually, 360° horizontal coverage with cameras is reached by mounting several 180° cameras on the vehicle. For ADAS, cameras are mounted around the body of the vehicle (front bumper, mirrors, trunk lid) while for self-driving they are mounted on the vehicle’s roof, e.g. at the roof edge centers.

Variations:

  • Cameras can by tilted to better cover the close environment around the vehicle.
  • Cameras can be mounted on the corners of the vehicle instead of the roof edge centers, or as a central “camera tower”, see below.

Pros

  • 360° coverage.
  • Enables top view to support parking.

Cons

  • 180° cameras are subject to severe lens distortion
  • The cameras resolution will be spread over a large area, leaving a comparably low resolution per degree that effectively limits the range of object detection.

There are several common ways of mounting cameras apart from the “cameras at the roof edge centers” approach described above:

Cameras at the roof corners

Cameras at the roof corners show larger areas of low sensor reliability compared to cameras on the roof edge centers (highly depending on vehicle dimensions). Additionally, the highly important view to the front is covered by the outer parts of the camera’s field of view; for minimum distortion, the view to the front would be on the optical axis of a camera.

Central camera tower

Instead of 180° cameras, 120° cameras that are less affected by lens distortion can be mounted as a “camera tower” on the roof center. This can be combined with a central 360° lidar to form on compact sensor module. As described for lidars above, the sensors mounted in this configuration can’t observe objects occluded by the roof.

Front camera

Front cameras are the primary (and often only) sensor for environment recognition in current series cars. They are comparably cheap but enable a variety of ADAS features – although they are limited by their sensibility towards environment conditions, like all cameras.

Variations

  • Front cameras are typically mounted between the rear mirror and the windshield. The windshield protects the camera and is cleaned by the wipers.
  • Alternatively, front cameras can be installed inside the vehicle between the dashboard and the windshield, or outside of the vehicle on the bumper or at the front roof edge center.
  • Stereo cameras can provide distance estimation.

Pros

  • Front cameras benefit from the vehicle’s front lights at night.
  • Enables ADAS features like lane departure warning, lane change assistant
  • If installed behind the windshield: protection from rain and dirt,

Cons

  • Vertical FOV is limited by vehicle’s hood hence small objects in front of the vehicle are occluded (unless camera is mounted on front bumper).
  • Insufficient overall coverage for urban AD if not combined with other sensors.

Wide angle camera for traffic light detection

Front camera and surround view camera systems are usually unable to detect objects the are close and elevated, such as traffic lights, because of their limited vertical FOV. This can be solved by adding a wide-angle camera mounted at the front roof edge center. Depending on the positioning of traffic lights in the target environment, it may be necessary to tilt these cameras upwards.

Alternative vision sensors

Apart from the setups based on mono cameras discussed above, there are several further interesting sensors for visual perception:

  • Stereo cameras provide distance estimation though 3D reconstruction.
  • Thermal cameras enable better detection accuracy at night and under challenging weather conditions.
  • High-framerate cameras drastically simplify object tracking and detection of moving objects.
  • Event-trigged cameras reduce the bandwidth needed for data transmission and enable a low-latency reactive path for collision avoidance.

Radar

Radars are already established in the automotive industry, they have been employed in series cars since many years to enable ADAS features such as Adaptive Cruise Control (ACC) and Autonomous Emergency Breaking (AEB).

Radars accurately measure distance and radial velocity. They are particularly good at detecting metallic objects but are also able to detect non-metallic objects such as pedestrians with reduced range.

360° coverage by short- and mid-range radars

Pros

  • A set of 4-6 short-range radars (SRR, ~30m range) and mid-range-radars (MRR, ~100m range) suffices to reach 360° horizontal coverage and solve many typical scenarios in urban ODDs.
  • Provides redundancy for lidar and camera while being more robust against challenging environment conditions.

Cons

  • The narrow vertical FOV of conventional automotive radars poses severe challenges when facing non-flat terrain.

Front radar

Pros

Cons:

  • Insufficient overall coverage for urban AD if not combined with other sensors.

Conclusion

Sensing for automated cars is like sensing for living beings – there is no one perfect sensor configuration that prevails in all environments and fits all budgets. The range of available sensor products is increasing fast, and there are multiple configurations that do the job, with their respective pros and cons.

Providing background on sensor set design and exploring frequent patterns for the different sensor modalities in this article did not aim for providing an absolute answer; instead it was supposed to convey the thought process of sensor set design, enabling an understanding of the rationale behind sensor choices.

If the concepts and patterns above intrigue you, a next step for getting more familiar with sensors for autonomous driving is to take a deeper look into sensor technology and the sensor products that are currently on the market. The next section lists helpful materials for this purpose.

Additional resources

Public sensor configurations

Some companies working on self-driving cars have already released their sensor configurations:

Sensor technology

Sensor specifications

 

Categories: general

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close