category: literaturenote citekey: ghasemieh3dobjectdetectionautonomous2022 title: "3D object detection for autonomous driving: Methods, models, sensors, data, and challenges" authors: "Ghasemieh, Alireza; Kashef, Rasha" year: 2022 date: 2022-06-00 06/2022 doi: 10.1016/j.treng.2022.100115 publication: Transportation Engineering url: "https://linkinghub.elsevier.com/retrieve/pii/S2666691X22000136" zotero_key: KBRUMLZW zotero_storage: BHWM7II6 collections: magistritöö / kohalikud teed folder: 001_artiklid firstAuthor: "Ghasemieh, Alireza"

status: converted

Contents lists available at ScienceDirect

Transportation Engineering

journal homepage: www.sciencedirect.com/journal/transportation-engineering

3D object detection for autonomous driving: Methods, models, sensors, data, and challenges

Alireza Ghasemieh ¹ , Rasha Kashef 2,*

Department of Electrical, Computer and Biomedical Engineering, Ryerson University, Toronto, Canada

ARTICLE INFO

Keywords: Autonomous vehicles Sensors LiDAR Point cloud Stereo images 3D object detection

ABSTRACT

Detection of the surrounding objects of a vehicle is the most crucial step in autonomous driving. Failure to identify those objects correctly in a timely manner can cause irreparable damage, impacting our safety and society. Several studies have been introduced to identify these objects in the two-dimensional (2D) and threedimensional (3D) vector space. The 2D object detection method has achieved remarkable success; however, in the last few years, detecting objects in 3D have received more remarkable adoption. 3D object recognition has several advantages over 2D detection methods, as more accurate information about the environment is obtained for better detection. For example, the depth of the images is not considered in the 2D detection, which reduces the detection accuracy. Despite considerable efforts in 3D object detection, it has not yet reached the stage of maturity. Therefore, in this paper, we aim at providing a comprehensive overview of the state-of-the-art 3D object detection methods, with a focus on 1) identifying advantages and limitations, 2) revelling a novel categorization of the literature, 3) outlying the various training procedures, 4) highlighting the research gap in the existing methods and 5) building a road map for future directions.

Introduction

The Society of Automotive Engineers (SAE) [1] defines Autonomous Driving (AD) systems in six levels (Fig. 1) based on the level of vehicle control and traffic condition.

Level 0 (No Driving Automation): The driver is responsible for all observations and actions
Level 1 (Driver Assistance): An advanced driver assistance system (ADAS) helps the driver in limited and specific functions. However, the human driver is still in charge of everything.
Level 2 (Partial Driving Automation): The driver's assistance takes control of the steering (wheel) and acceleration under specific conditions like dring on highways only, but the human driver's attention is needed in an emergency.
Level 3 (Conditional Driving Automation): All functions are shifted to the vehicle, but the human driver interference could be done if necessary.
Level 4 (High Driving Automation): This is a fully autonomous system responsible for all functions during a trip in an "operational design domain (ODD)"
Level 5 (Full Driving Automation): This level of an autonomous system is expected to operate as well as a human driver and handle all situations that a human driver can tackle.

An autonomous vehicle (AV) combines three fundamental systems: Perception System, Planning System and Control System [3]. The perception system is responsible for Environmental Perception and Localization to help two other systems function correctly and reliably. Environmental information is received through various sensors installed around the front and top of the car. The received information is processed by machine learning (e.g. supervised and unsupervised deep learning) methods and turned into semantic information. Fig. 2 shows a typical autonomous vehicle system. Object detection is one of the primary activities of the perception system, which detects the class of an object like pedestrians, cycles and cars, its pose, and its orientation if moving. This information is required for planning, and control systems.

https://doi.org/10.1016/j.treng.2022.100115

Received 3 March 2022; Received in revised form 14 April 2022; Accepted 15 April 2022

Available online 20 April 2022

^* Corresponding author.

E-mail addresses: alireza.ghasemieh@ryerson.ca (A. Ghasemieh), rkashef@ryerson.ca (R. Kashef). 1 Alireza Ghasemieh is a graduate student at Ryerson University, Toronto, ON CANADA 2 Dr. Rasha Kashef is a faculty member at Ryerson University and the director of the IOTA lab in the Department of Electrical, Computer, and Biomedical Engineering

Fig. 1. Autonomous Driving levels [2].

Fig. 2. Typical autonomous vehicle system [3].

Table 1 Advantages and disadvantages of different sensors.

Sensors	Advantage	Disadvantage
Monocular	• RGB image, texture	• No depth information
Camera	attributes	• Susceptible to weather and light
	• Not expensive	conditions
Stereo	• RGB image, texture	• Susceptible to weather and light
Camera	attributes	conditions
	• Not expensive
LiDAR	• Accurate depth information	• Expensive
	• Less susceptive to weather	• No texture attribute
	conditions
	• Not susceptive to light
	condition
	• Panoramas observation

There are two categories of Object detection: two-dimensional (2D) object detection and three-dimensional (3D) detection. 2D methods are well addressed by several works in both classification and localization. However, they do not detect objects' depth information, which is essential for planning. In these methods, objects are localized by a 2D

• No colour information Fig. 3. IMX390 LED flicker mitigation [6].

boundary box with no depth dimension.

The 3D detection methods provide a third dimension that offers more information about an object's depth, size, and location. These methods draw a 3D bounding box around each object, and some of them also detect the object's orientation if moving. Despite 2D methods, 3D object detection methods are not mature yet and need refinement to enhance the models' detection accuracy or lower the computational complexity and cost. In 3D object detection, monocular and stereo image with a 3D

Fig. 4. IMX390 leaving tunnel light saturation [6].

Fig. 5. LiDAR point cloud output.

point cloud is the standard layout that provides depth information about observed objects. There are no standards for using these two data sources from cameras and LiDAR. Furthermore, the alignment of multiple cameras (front, sides, bumpers) and LiDAR bird's eye is a big challenge in 3D object detection. Various methods utilize this information in different ways, which cause sparsity and irregularity in this field.

Statement of Differentiation: In recent years (2015 – present), notable techniques and procedures have been presented to address 3D object detection. However, no research work provides a comprehensive overview of the state-of-the-art methods from the modelling, categorization, analysis, and applicability. Thus, in this paper, the main contributions of this paper are to 1) focus on introducing a systematic review of the most recent advances in 3D object detection methods, including the fusion of modalities methods, 2) address the key novelty of each technique, summarize the training procedure through the proposed network architecture, 4) conduct a comparative analysis to discuss the pros and cons of each method.

This paper is organized as follows: in section 2: sensors used to gather environmental information are presented, section 3 describes commonly used datasets to train and test the models, section 4: introduces the simulations that provide datasets instead of using real data captured from a vehicle, section 5: the 3D object detection techniques are discussed in detail, summarization discussions and conclusion are presented in sections 6 and 7, respectively.

Sensors

Generally, there are two types of sensors: Passive and Active sensors. Passive sensors receive the natural emission from an environment, such as natural light or infrared rays. The camera is the most commonly used passive sensor in a car that captures a 2D array of coloured points through a lens and sends it to the AD system. Active sensors send signals and receive the reflection from surfaces and objects. LiDAR is a commonly used active sensor in a car. This class of sensors is more expensive than passive sensors, but they can measure the depth of objects essential for the AD system by sending and receiving reflected signals.

Regarding certainty, as we mentioned using cameras only is not sufficient. A complimentary system like LiDAR is needed to compensate camera functions in adverse weather or low lumination to ensure the AD detects all objects. Combining the output of different types of sensors is called multi-view fusion. Regarding reliability, all sensors should have a backup in case of failure or outage to ensure that if a sensor malfunctions, another one replaces its function immediately, not jeopardizing the car occupants' safety. Table 1 summarizes the advantage and disadvantages of each sensor.

Camera

Cameras provide environmental information about objects, shapes, and texture properties. This information is used to classify [4] and, locate objects, detect lane geometry and traffic signs around a car [5]. The camera is usually installed behind the front mirror and captures the front scene of the vehicle. Since the camera is not expensive, many cars have at least one installed. Some modern cars have two cameras on the front bumper to enhance the vision of the AD system; others have sides cameras or an array of cameras on the top. As we can see, there is no standard in camera installation and usage, and based on this diversity, various number object detection methods are proposed; all methods use at least the front camera for object detection.

Generally, camera performance is highly susceptible to weather and

Fig. 6. CARLA simulator environment [18].

Table 2 Summary of public dataset features.

Dataset	Year	Scenes	Classes	Size	Frames	3D Boxes	Cities	Monocular Image	Stereo Image	360 Cameras	LiDAR	GPS	RADAR	IMU
KITTI [11]	2012	50	8	43.67	15K	200K	5011	Yes	Yes	No	Yes	Yes	–	–
nuScenes [12]	2019	1K	23	75.75	40K	1.4M	225	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Waymo [13]	2020	1K	4	336.62	200K	12M	31	Yes	Yes	Yes	Yes	–	–	–
Lyft Level 5 [14]	2019	1805	9	102.34	46K	1.3M	–	Yes	Yes	Yes	Yes	–	–	–
H3D [15]	2019	160	8	86.02	17K	1.1M	31	Yes	Yes	No	–	–	–	–
Applloscape [16]	2019	103	26	206.16	100K	–	78	Yes	Yes	No	–	–	–	–
Argoverse [17]	2019	113	15	97.81	44K	993K	88	Yes	Yes	Yes	–	–	–	–

Table 3 Comparison of 3D object detection methods.

Category	Subcategory	Model Name	Modality	Num of Stage	Dataset	Support Occluded Object
Monocular/ Stereo Image	Template Matching	3DOP	Stereo Image	2	KITTI	No
		Mono3D	Monocular Image	2	KITTI	Yes
		Deep MANTA	Monocular Image	2	KITTI	Yes
		3DVP	Monocular Image	2	KITTI	Yes
		SubCNN	Monocular Image	2	KITTI	Yes
	Geometric Properties	Deep 3DBox	Monocular Image	2	KITTI	Yes
		GS3D	Monocular Image	2	KITTI	Yes
		Stereo R-CNN	Stereo Image	2	KITTI	Yes
Point Cloud	Multi-view	VeloFCN	LiDAR	1	KITTI	Yes
		PIXOR	LiDAR	1	KITTI	Yes
	Voxel Grid	VoxelNet	LiDAR	1	KITTI	NA
		SECOND	LiDAR	1	KITTI	Yes
		PointPillars	LiDAR	1	KITTI	NA
		Part-A2 Net	LiDAR	2	KITTI	Yes
		HVNet	LiDAR	1	KITTI	Yes
	Point Set	PointRCNN	LiDAR	2	KITTI	Yes
		3DSSD	LiDAR	2	KITTI, nuScenes	Yes
		PointGNN	LiDAR	1	KITTI	Yes
	Point Voxel Hybrid	Fast Point R-CNN	LiDAR	2	KITTI	Yes
		STD	LiDAR	2	KITTI	Yes
		PV-RCNN	LiDAR	2	KITTI, Waymo	Yes
		SA-SSD	LiDAR	1	KITTI	Yes
Fusion	Parallel Methods	MV3D	Image, LiDAR	2	KITTI	NA
		AVOD	Image, LiDAR	2	KITTI	NA
		Frustum PointNet	Image, LiDAR	2	KITTI	NA
		Multi-Level Fusion	Image, LiDAR	2	KITTI	NA
		Frustum ConvNet	Image, LiDAR	2	KITTI	NA
		MVF	LiDAR	2	KITTI	NA
		UberATG-MMF	Image, LiDAR	2	KITTI	NA
		MVX-Net	Image, LiDAR	2	KITTI	NA
		LaserNet	LiDAR	2	KITTI	NA
		LaserNet++	Image, LiDAR	2	KITTI	NA
		3D-CVF	Image, LiDAR	2	KITTI	NA
		CLOCs	Image, LiDAR	2	KITTI	NA
		RVF-Net	Image, LiDAR	2	nuScenes	NA
		CenterFusion	Image, LiDAR	2	nuScenes	NA
		SLS-Fusion	Image, LiDAR	1	KITTI	NA
		Cross-Modality	Image, LiDAR	2	KITTI	NA
	Sequential Methods	UberATG-ContFuse	Image, LiDAR	2	KITTI	NA
		Pseudo-LiDAR	Image	2	KITTI	NA
		Pseudo-LiDAR++	Image, LiDAR	2	KITTI	NA

		E2E Pseudo-LiDAR	Image	2	KITTI	NA
		PointPainting	Image, LiDAR	2	KITTI	NA

Fig. 7. Left to right: original image, stereo reconstruction, depth-based features, prior [24].

light conditions. Foggy, rainy or snowy weather reduces vision radius and drastically decreases object detection accuracy and reliability. Furthermore, light change from sunny day luminary to night-time or extreme brightness disparity that causes image degradation is a big challenge. For instance, entering a tunnel reduces the light rapidly that blinds the camera vision for a few seconds. Leaving the tunnel when the outside is extremely bright is challenging for the AD system. Another example is LEDs used in traffic signs and vehicle brake lights that cause a flickering problem. Cameras can not accurately capture environmental information due to LED switching behaviour. Sony makes some effort to reduce this issue by developing IMX390 to mitigate the flickering problem [6]. Fig. 3 and Fig. 4 show the LED flicker and car leaving a tunnel using an IMX390 filter to overcome the flickering issue.

Mainly two types of cameras are used in AV: Monocular and Stereo Camera. Monocular cameras provide 2D array pixels that contain detailed information about the car environment. One disadvantage of

this type of camera is the lack of depth detection capability, and this information is used to determine the object's size and location on the surface. On the other hand, stereo cameras have two or more lenses with separate image sensors to extract depth information, which requires more processing power [7].

Light detection and ranging (LiDAR)

LiDAR is an active sensor that scans the environment by emitting a laser beam, receiving the reflection, and measuring each pulse's travel time to extract depth information. The output of LiDAR is a 3D point called Point Cloud (PCL). As Fig. 5 shows, the output of LiDAR is a map of sparse points that does not have a uniform distribution. A disadvantage of the point cloud is not capturing the texture and colour of objects and surfaces used in semantic detection. However, having sparse points help to reduce the storage size and fasten the process. Since LiDAR is an

Fig. 8. Mono3D overview approach [25].

Fig. 9. Overview of Deep MANTA approach [26].

active sensor, it is less susceptible than cameras and is more reliable under different weather and lumination conditions. However, its performance and accuracy decrease adversely (e.g. rainy and snowy). Moreover, It should be mentioned that this type of sensor is costly.

Typical standard LiDAR systems like HDL-64E have a rotating laser beam that scans the environment 360◦ with 0.3◦ resolution horizontally and 27◦ with 0.4◦ resolution vertically up to 120 m radius with an accuracy of 2 cm. [8] Some extended models like VLS-128 can cover up to 300 m [9].

Datasets

Many object detection models have been developed. To compare their performance, there are some well-known datasets that researchers have used to evaluate their result and compare it with others. The bestknown dataset in AD is KITTI, generated by the Karlsruhe Institute of Technology (KIT) and Toyota Technological Institute at Chicago (TTI-C) [10]. The KITTI dataset is generated by two high-resolution colour and grayscale video cameras, a Velodyne laser scanner, and a GPS localization system. The interest to use this dataset include stereo images, optical flow, visual odometry, 3D object detection and 3D tracking. The dataset contains the stereo-coloured image, LiDAR point could and GPS coordinate synchronized in time. KITTI contains different scenes like urban areas, well-structured highways, downtown streets and countryside roads categorized by difficulty to detect as easy, moderate and hard [11]. One disadvantage of this dataset is that all the recordings are down during sunny weather conditions and do not include rainy, snowy, foggy weather conditions. The lumination ranges from extreme brightness to night-time darkness. Table 2 summarizes the available public datasets designed explicitly for autonomous deriving purposes. Despite several datasets, all empirical experiments were conducted and evaluated on the KITTI dataset.

Fig. 10. 3D Voxel Patterns [27].

Fig. 11. SubCNN network overview [28].

Simulations

Besides real-world datasets, two well-known 3D open-source simulators for AD are used in model training and testing: CARLA and Sim4CV. These simulators have 3D pedestrians, cars, trucks, buses, and cycles that generate real situations like object occultation and truncation. As Fig. 6 shows, CARLA includes two virtual towns allowing environment and sensor configuration. An advantage of this simulator is that it can add different weather conditions and lumination to create unseen scenarios. It has LiDAR with a monocular/stereo camera [18]. Sim4CV is a photo-realistic simulator that gives bounding boxes as ground truth and a flexible environment to produce different scenarios. [19]

Object detection methods

There are variant object detection methods. However, we focus only on 3D object detection methods. These methods can be divided into three categories regarding the modality of input data: Monocular/Stereo Image-Based, Point Could-Based and Multimodal Fusion-Based methods. Each category could be subdivided; Table 3-7 compare these categories.

Monocular/Stereo image-based methods

Monocular cameras provide RGB images from the AV environment, and this image is used to predict 3D object bounding boxes. These methods are similar to 2D object detection with further processing to extract the depth from a 2D image. In this category, we can split the research into two lines: Template Matching-based methods and Geometric Properties-based methods.

Region proposal is a basic block of these methods responsible for proposing the region of an object in the image. In this way, the amount of processing to detect an object reduces dramatically [20]. There have been four significant methods for region proposal: traditionally handcrafted grouping [21], Selective Search [21], window scoring [22] and the latest method Regional Proposal Network (RPN) [23]

Template matching-based methods

These methods can perform 2D/3D matching through comprehensive sampling and scoring 3D suggestions as demonstration patterns.

3DOP. The early template matching-based method is known as 3DOP by Chen et al. [24], which proposes 3D object detection using stereo images as input to extract depth information and build a point of cloud set. Later the set is reprojected to 3D image planes to provide a 3D space. 3DOP uses the energy minimization of Markov Random Field to propose 3D objects, and it uses Fast R-CNN to regress the location of objects. Fig. 7 shows the steps to create 3D space from stereo images.

Mono3D. If one camera does not see an object in both cameras because of any obstacles or having a monocular camera, Chen et al. advanced the 3DOP by using a sliding window to propose 3D objects directly without computing the depth and called it Mono3D. The assumption is ground plane is orthogonal to the image plane to reduce the search. Mono3D, like 3DOP, uses Fast-R-CNN to output class-specific proposals [25]. Fig. 8 shows an overview of the Mono3D approach.

Deep MANTA. Deep MANTA is another template matching-based method proposed by Chabot et al. They used an extensive CAD 3D models library to match the 2D bounding box of the detected 2D objects with 3D models. Deep MANTA scores the all proposed 3D template to find the best match of object geometry in the database. The system has two main steps. First, the monocular image passes through the Deep MANTA network to generate a bounding box with a matching vehicle geometry score. The second step is the inference that the outputs and 3D models are used to extract 3D orientation and locations [26]. As Fig. 9 shows, the first step has three levels. The first level produces a region proposal using RPN. Level 2 extracts the bounding box properties; level 3 provides the class of object, 3D part coordinate, part visibility and a 3D model template.

Voxel pattern (3DVP). In crowded places like urban areas, many objects block the view of others that only a portion of an object is visible to cameras and the rest is occluded or even truncated. Fig. 10 shows an instance of occluded and truncated objects. This phenomenon causes a severe issue in object detection and estimating its 3D properties. Xiang

Fig. 12. GS3D process [30].

Fig. 13. Stereo R-CNN Network architecture [31].

Fig. 14. Point cloud calculated from the LiDAR. (a) Perspective view of the point cloud (b) BEV (c) front view [32].

et al. [27] propose a novel method to extract the object's properties (3D shape, viewpoint, occlusion and truncation). This method indicates which part of an object is visible, occluded or truncated. They use a dictionary of 3DVPs and train specialized detectors to detect objects in a 2D image and use meta-data to transfer and match them with 3DVPs. This method improves the object detection in KITTI by 6% under challenging samples.

SubCNN. 3DVP depends on the performance of the Region Proposal

Network (RPN) and fails in detecting occluded, truncated or scaled objects. Xiang et al. improved the 3DVP method to overcome this issue and proposed Subcategory-Aware CNN (SubCNN) [28]. They use subcategory information to help RPN with the proposal generating process. In other words, they use CNN ad RPN level, which increases the computational cost but increases the RPN performance. Fig. 11 shows the SubCNN overview.

Geometric properties-based methods

These methods use geometric properties to predict 2D bounding boxes instead of extensively searching to generate proposals with a high recall rate.

Deep3Dbox. Mousavian et al. [29] proposed the Deep3Dbox method that uses a CNN to regress 3D object properties and then incorporates a 2D bounding box to limit the object's geometry over the 3D bounding box. The first network detects the 3D orientation, and the second network regresses the 3D bounding box. The advantage of this method is the 3D orientation detection using a novel hybrid discrete-continuous loss which is more effective than L2 loss.

GS3D. Li et al. proposed [30] to detect the 3D bounding box from 2D RGB images without using labels. First, it uses a 2D+O subnet to get a 2D bounding box and orientation. It uses Faster R-CNN to detect the orientation. Then based on the output, it generates a coarse cuboid for each predicted 2D box utilizing projection relation between 2D and 3D. This 3D box has enough accuracy to refine it later. The next step is feature extraction from the 2D bounding box and finally does refinement on the 3D bounding box to fit it into the object using a 3D subnet. Fig. 12 presents the GS3D overall procedure.

Stereo R-CNN. Li et al. proposed the Stereo R-CNN [31], which uses stereo images to detect 3D objects. This method extends the Faster R-CNN method to use stereo images. Like other methods, it uses RPN to get proposal regions, and by adding extra branches, it can predict sparse key points, viewpoints, and object dimensions. This method combines left and right 2D RGB images to calculate an accurate 3D bounding box without depth information. This method performs better than previous methods by around 30% AP on 3D object detection and localization over the KITTI dataset. Fig. 13 presents the network architecture.

Point cloud based methods

As we mentioned, the point cloud is the output of LiDAR. This output

Fig. 15. Overview of PIXOR network [34].

Fig. 16. Four types of 3D object presentations [35].

is sparse, and the goal is to learn from these sparse and irregular points. Irregularity in the domains is an issue for CNN learning. Therefore, using the convolution kernel directly on the point cloud results in inappropriate learning, and it means a different type of learning or preprocessing techniques are required. The learning methods based on point cloud can be categorized as Multi-View, Voxel Grids, Point Set and Point Voxel Hybrid Methods.

Multi-view-based methods

The easiest way to deal with the sparsity of point cloud is converting the view to either Bird's Eye View (BEV) or Front View, which are density in grids. Fig. 14 [32] shows the perspective view (a) and BEV (b), and the front view of the output of LiDAR.

VeloFCN. Li et al. [33] proposed to convert the point cloud to the front view presentation and then used a fully convolutional network to detect the objects and regress the 3D bounding box. They used the KITTI dataset to test the performance. This method only detects vehicles and has limited performance on occluded and small objects.

PIXOR. Yang et al. [34] proposed a single-stage proposal-free and real-time (10 FPS) 3D object detection that, compared to the previous methods, needs lower computational power and can be used in real-time applications. They converted the point cloud to BEV and then used five blocks of ResNet in the backbone of the PIXOR detector called ResNet-LiDAR with 55% Average Precision (AP). This method can not detect any pedestrian, traffic sign or occluded object using a BEV map. Fig. 15

Voxel grids methods

This type of method relies on the Volumetric presentation of a 3D object. Fig. 16 shows the four commonly used 3D object presentations. A volumetric presentation is a discretization form of the point cloud that degrades the object's localization accuracy, and the voxel methods based on it are computationally more efficient. Volumetric presentation is a many-to-one mapping process called hard voxelization. In this

Fig. 17. VoxelNet network architecture [36].

Fig. 18. VFE layer [36].

Fig. 19. SECOND detector structure [37].

Fig. 20. Overview of PointPillar network [38].

method, 3D CNNs are used to extract the object features.

VoxelNet. Despite previous methods which conduct handcrafted feature representations for point cloud, like BEV projection, VoxelNet [36] proposes a generic single-stage and end-to-end trainable deep network to extract features and regress the 3D bounding box. Zhou et al. also proposed a novel voxel feature encoder (VFE) (Fig. 18) to divide the point cloud into an equally 3D voxel space and then transform and unify a group of points into the feature representation. Fig. 17 shows the VoxelNet architecture. It learns feature representation within three steps: 1) Feature Learning Network, 2) Convolutional Middle Layer, 3) Region Proposal Network.

The first component divides the point cloud into equally spaced 3D

voxels, then VFE transforms a set of points into a vector that determines the shape of the object's surface later. The convolutional layer is used to get the contextual shape of the voxel-wise features. Finally, the RPN uses the 3D CNN output to predict the objectness and the boundary box. As it uses a 3D CNN and multiple feature extraction layers, the computational cost of this method is high.

SECOND. As mentioned, VoxelNet's main drawback is cubically computational complexity, which results in high memory consumption needs. Yan et al. proposed Sparsely Embedded Convolutional Detection (SECOND) [37] to resolve VoxelNet computational issue. Instead of using a fully connected network, SECOND uses spare convolutional operation, reducing memory consumption and increasing

Fig. 21. Part-A2 network architecture [39].

Fig. 22. Structure of HVNet [41].

computational speed. As Fig. 19 presents, it first records the voxel coordinates and the number per voxel. Then they use a voxelwise feature extractor with a fully connected network (FCN)to extract point-wise features. Later, a sparse convolutional middle extractor decreases the computational complexity and speeds up the training and inferencing. But despite that, SECOND eliminates the unnecessary computation squandered by unavailing zero-padding voxels.

PointPillars. Lang et al. [38] consider that either a model uses a form of point cloud representation to fasten the process by sacrificing the accuracy or employs the data to have better accuracy but being slower. They propose a novel encoder called PointNets to learn a representation of point clouds organized in vertical columns (pillars). PointNet can be used with any 2D CNN detector. The advantage of using this method is having an accurate and fast detector. They implement this method on the KITTI dataset. The results show the performance of running achieved 63 FPS by getting rid of 3D CNN used in VoxelNet, which is faster than

previous point cloud-based models and fusion-based models. As Fig. 20 shows, the PointPillar consists of three main components: 1) Pillar Feature Net (feature encoder), 2) 2D convolutional network, and 3) A detection head. The first component converts point clouds to vertical pillars, maps them to BEV perspective and creates a pseudo-image. The 2D convolutional backbone is similar to a U-Net that downsamples the pseudo-image and then upsamples and concatenates at the second stage.

Part-A2 net. Shi et al. proposed Part-A2 [39] by extending their previous work PointRCNN [40]. The main idea is intra-object part locations that a 3D bounding box can provide without occlusion issues. The framework has two stages: the part-aware stage and the part-aggregation stage. The first stage has a U-Net-like architecture that encodes a sparse convolutional network and then decodes by using free-of-charge part supervision extracted from 3D ground-truth box, which helps accurately predict the intra-object part location. The second stage is responsible for re-scoring and refining the box properties by extracting the relationship

Fig. 23. PointRCNN network architecture [40].

Fig. 24. 3DSSD network architecture [44].

Fig. 25. GNN steps [46].

of the intra-object part locations. This stage reduces the ambiguity of the 3D bounding box and learns the relationship of points within the 3D proposal resulting from the first stage's anchor-free or anchor-based.

Fig. 21

HVNet. Voxelization shows a promising form of point cloud representation. However, there is an issue when dealing with different sizes of objects. The scall of an object directly affects the accuracy of 3D object detection; therefore, the size of voxels is a parameter for modelling. A

smaller voxel performs better, especially for small objects, but has a longer inference time. A more extended voxel size has lower performance with a small object but is faster processed. Ye et al. [41] propose a hybrid voxel network (HVNet) to address this issue. This method uses a multi-scale level of point-wise voxel feature encoder (VFE), fuses them to get contextual objects, and then maps the meta-data to the original voxels to localize the object accurately. Fig. 22 top line shows the overall procedure of HVNet.

The bottom line shows the Hybrid VFE architecture, which includes

Fig. 26. PointGNN network architecture [45].

Fig. 27. Fast Point R-CNN network architecture [47].

four parts: 1) Multi-scale pre-voxelization; 2) Multi-scale point cloud data encoding; 3) Encoded features aggregation; 4) Dynamic feature projection

Point set methods

These methods use raw point cloud with PointNet++ [42] and Graph Neural Network (GNN). They usually keep the geometry of points, and retrieving in 3D space is inefficient like volumetric-based methods. Learning 3D representations directly from point cloud raw data is feasible using PointNet [43] that Qi et al. proposed. In this method, converting the point cloud into volumetric grids is unnecessary. The improved version, PointNet++ [42], is used in PointRCNN [40] and 3DSSD [44].

PointRCNN. PointRCNN's [40] main idea is to expand Faster RCNN [23] from 2D space to 3D space. Shi et al. proposed this framework to utilize the point cloud raw data. As

Fig. 23 shows, The method has two stages: 1) bottom-up 3D proposal generation and 2) refining proposals in canonical coordinate. The proposals are generated using the semantic segmentation of points into background and foreground using PointNet++ architecture. The second stage transforms the 3D proposed region pooled points to canonical coordinates to learn local spatial features more accurately. Its limitation is its inference time which both steps are time-consuming.

3DSSD. Point set-based methods are slow due to using a feature Propagation (FP) layer for broadcasting features to the whole scene via upsampling. Yang et al. [44] proposed 3DSSD to reduce inference time consumption by eliminating the FP layer. Instead, they used Euclidian and feature metrics when performing Furthest Point Sampling (FPS). Multiple set abstraction (SA) layers are used to downsample and extract context features in the first stage. Next, FP layers are used for upsampling and broadcasting features to the points. Later the RPN layer is applied to generate region proposals. In the second stage, a refinement

Fig. 28. STD network framework [48].

Fig. 29. PV-RCNN network architecture [49].

module gives the final predictions. The empirical result showed that 3DSSD is two times faster than PointRCNN with 25 FPS, which is good enough to use in real-time applications.

Fig. 24

Point-GNN. Graph Neural Network has a great reasoning ability over non-Euclidean data like point cloud and is widely used for semantic segmentation. Its capability can be improved to enhance the 3D object detection accuracy. Shi et al. propose Point-GNN [45] using GNN to represent point cloud data. In this method, as Fig. 25 shows, GNN connects neighbour points (vertex) within a radius through edges to encode the point cloud to a graph network and updates the connections over some iterations to refine them without any quantization. The update process detects and localizes 3D objects and then merges a 3D bounding box to fit it to a detected object. The main issue of this method is the computational complexity that over KITTI took over a week to train the model.

Fig. 26

Point voxel hybrid methods

This class of methods takes the benefits of both voxels and raw data.

The voxel resolution is highly correlated with the Voxel-based localization and detection accuracy with computational complexity. On the other hand, point-based methods keep the locality of points and can either use SA or PointNet to provide neighbour information. Therefore, having both approaches in a method could be helpful.

Fast point R-CNN. Chen et al. introduced the Fast Point R-CNN [47], taking advantage of volumetric representation and raw point cloud data. As Fig. 27 shows, it is a two-stage method; the first stage provides a few high-quality initial predictions using convolutional operations over a voxel representation of the input. The second stage works on interior predictions points to refine and enhance each object's localization and context information. The empirical experience over KITTI 3D and BEV detection shows this method can reach 15 FPS with better accuracy in BEV moderate class than previous methods.

Sparse-to-Dense (STD). Yang et al. presented the STD [48] in two stages. As Fig. 28 shows, the first stage generates accurate proposals from raw point cloud data using a bottom-up network which seeds a spherical anchor for each point and provides higher recall with lower computational cost. The second stage is responsible for box regression, and it uses

Fig. 30. The SA-SSD network architecture [50].

Fig. 31. MV3D network architecture [51].

the parallel Intersection-over-Union (IoU) branch to enhance the localization accuracy. They conducted the training and evaluation over the KITTI dataset, and the results showed that 3D and BEV detection outperformed on the hard sample set by 10+ FPS.

PointVoxel-RCNN. Shi et al. proposed PV-RCNN [49], which deeply integrated 3D voxel CNN and PointNet-based SA. PV-RCNN take benefits from flexible receptive fields of PointNet-based network and the accuracy and computational efficiency of voxel-based 3D proposal network. It summarizes the whole scene with a sparse 3D convolution to a set of key points via a novel voxel set abstraction (VSA) to save follow-up computations. RoI grid pooling is proposed abstract proposal-specific features from the key points to the RoI-grid points via keypoint SA using the highly accurate 3D proposal generated by voxel CNN. This method is tested over KITTI and Waymo open datasets, and the result showed it outperformed the existing methods on Car 3D detection by a large margin.

inference stage. Fig. 30

SA-SSD. Point cloud downsampling loses spatial information and decreases the localization performance. To address this issue, He et al. proposed Structure Aware Single-stage 3D Object Detection (SA-SSD) [50], which is a one-stage object detector. They enhance the localization precision by leveraging the structure information of point clouds. They introduce an auxiliary network that helps to convert the convolutional feature back to point-level representation. After the training step, the auxiliary network is no longer needed and can be detached from the pipeline to reduce the computational complexity during the inference stage. They also developed a part-sensitive warping operation to refine the 3D bounding box that usually suffers from the discordance between predicted bounding boxes and associated classification confidence. The method is tested over KITTI 3D/BEV test dataset, and the empirical experiment showed outstanding performance with 25 FPS during the

Fig. 32. AVOD network architecture [52].

Fig. 33. Frustum PointNet network architecture [54].

Fig. 34. Multi-level fusion from the monocular image [55].

Fusion methods

Fusion-based 3D object detection methods can be categorized by how inputs are combined. Therefore, we can group them into parallel, sequential, and hybrid combinations.

Parallel fusion methods

In the category, multi inputs are fed into the model in parallel. It means a separate pipeline for each input to carry on the processes.

Fig. 35. The sequence of frustums generated from a region proposal.

MV3D. Chen et al. [51] proposed Multi-View 3D networks (MV3D), a sensory fusion method that uses point cloud and image as its input and predicts the orientated 3D bounding box. MV3D has two networks: a 3D proposal network that gets LiDAR bird view, LiDAR front view and RGB image as the inputs, uses a 16-layer VGG net with some modifications and generates accurate prior 3D bounding boxes. Moreover, a region-based deep fusion network combines features from multiple views and determines the class of objects and the orientation of 3D bounding boxes. They used 2x bilinear upsampling on the point cloud dataset to enhance bird view's small object detection accuracy, resulting in a real-time and low-memory footprint object detection algorithm.

Fig. 31

AVOD. Ku et al. [52] proposed Aggregate View Object Detection (AVOD) to solve the bounding box orientation issue and the lack of enough information to generate region proposals for small objects. AVOD uses point cloud and RGB image as inputs and has two subnetworks: a region proposal network (RPN) and a detector network. They developed at the RPN stage a novel feature extractor using feature pyramid networks (FPNs) [53] idea to generate a high-resolution feature map to localize small objects. Moreover, they propose feature fusion RPN that uses multiple modalities to have highly recall RP for small objects. The second subnetwork uses these proposals to regress the orientated 3D bounding boxes accurately and predict the class of objects. The feature extractors include an encoder with modified VGG-16 with half the number of channels to perform 8x downsampling. A bottom-up decoder that learns to upsample the feature map using Feature Pyramid Network (FPN) to its original size. AVOD uses 1 × 1 convolutional layers to reduce the dimensionality anchors to save GPU memory from 5GB to a few Megabytes, which allows this method to be used in real-time application testing over KITTI.

Fig. 32

Frustum PointNet. Qi et al. [54] proposed a novel method to combine depth information from the point cloud with RGB images to detect 3D objects. They designed three networks: Frustum proposal, 3D instance segmentation and amodal 3D box estimation. The first step extracts the 3D bounding boxes using a well-performing 2D bounding boxes regressor and then extrudes it. In this way, they reduce the search space. They trimmed 3D space for each frustum and aligned point clouds into a canonical. This alignment results in a better projection of 3D geometry patterns.

In the second step, they used point cloud in frustum to segment the instance and finally amodal 3D bounding box regression at the following step. These features result in more accurate detection testing over the COCO datasets and later over KITTI datasets with fewer memory footprints. Moreover, it allows learning in 3D space, which preserves the 3D space geometry better.

Fig. 33

Fig. 36. Frustum ConvNet network architecture [56].

Fig. 37. The MVF network architecture [57].

Fig. 38. Conceptual relation between tasks and sensors [58].

Multi-level fusion from monocular image. Xu et al. [55] proposed a 3D object detection method based on monocular images with depth information from LiDAR data. It has a 2D proposal network and a 3D regression/classification network. The first network is responsible for generating proposals from 2D images, and it uses depth information from point cloud data. They used a novel method in localization which, instead of regressing the 3D coordinates (X, Y, Z) of the object centre, estimated the disparity from the typical dimension of objects in each class. This typical dimension is extractable from training data. The depth information is contaminated with the image to enhance the Conv layer performance. The output is augmented with point cloud region-wisely, and then an RoI Mean Pooling layer is implemented to convert the augmented point cloud in the proposal into a fixed-length feature vector. The second network is responsible for regression and classification of 2D and 3D bounding boxes, orientation, object classification and localization through Fully Connected (FC) layers. The point cloud features are concatenated with 2D proposals, and within FC layers, the properties above are generated.

Fig. 34

Frustum ConvNet. Wang et al. [56] proposed a three steps 3D object detection method using a 2D region proposal for RGB images at the first stage and a region proposal to generate multi-level frustums by sliding along the frustum axis to identify the 3D points corresponding to pixels

Fig. 39. The Network architecture [58].

Fig. 40. MVX-Net PointFusion network architecture [59].

Fig. 41. MVX-Net VoxelFusion network architecture [59].

inside each 2D proposal (Fig. 35).

Then F-ConvNet uses PointNet to aggregate point-wise features in the form of vector. Next, the obtained vectors at the early stage are concatenated to build a 2D feature map. The generated frustum 2D feature map is down-sampled and up-sampled by the fully convolutional network (FCN) to fuse all the features across the frustum axis. Finally, the 3D bounding box orientation is estimated in each proposal. The authors tested the method on SUN-RGBD and KITTI datasets and showed the method's superiority amongst existing methods.

Fig. 36

Multi-view fusion (MVF). Zhou et al. [57] proposed MVF, which only

uses LiDAR point cloud as input but in two different formats: BEV and perspective view. In this way, they added complementary information per-point embeddings. The algorithm is designed to fuse at an early stage in which the point-level presentation is still preserved, and the embedding method is different for each view. Vertical column voxelization [38] is utilized for BEV encoding and 2D Conv tower for perspective embedding on the "range-image-like" feature map. In this way, they reduced the sparsity issue of the point cloud dataset. The generated content features from a point and its neighbours in both views are infused and concatenated at point-level, which builds the final voxel-level embeddings. The second contribution is proposing a dynamic voxelization (DV) which obviates the need of having a fixed number of points to build a voxel, padding the voxel to a fixed size, and

Fig. 42. LaserNet network architecture [60].

Fig. 43. LaserNet++ network architecture [61].

stochastic dropout of point and voxels result in a more stable detection. They evaluated MVF over the KITTI dataset and proved that MVF could be incorporated with other LiDAR methods to enhance the detection capability.

Fig. 37

UberATG-MMF. Liang et al. [58] proposed a multi-task fusion model (UberATG-MMF) that benefits from multi-task learning, LiDAR point cloud, and RGB image fusion. The main prominent feature of previous models is defining multi related tasks: 2D and 3D object detection, depth completion and ground estimation (Mapping), which all can jointly help each other by fusing the information at different levels to refine the learning of feature presentations which result in better object detection performance. Therefore a loopback network is designed to implement the idea. Fig. 38 presents the idea concept: the relation of sensors at the bottom and between the tasks at the top. Moreover, the model can be learned end-to-end and handle all tasks simultaneously, making it suitable for real-time applications.

In order to take advantage of having fully fused feature representations, the network utilizes both point-wise and ROI-wise feature fusion. Since cars, cycles and pedestrians are located on the ground, knowing the ground location can help the model to detect 3D objects better. Therefore, the model estimates the gound location voxel-wisely, which

the BEV network uses to estimate relative location. Besides 2D and 3D object detection tasks, the model performs the task of depth completion to facilitate the learning through cross-modality feature representation and uses the pseudo LiDAR point in dense point-wise fusion with the image and LiDAR point cloud. Fig. 39 illustrates the explained procedure.

MVX-Net. Sindagi et al. [59] claimed that previously proposed models are either complex pipelines or perform late-fusion and are incapable of learning the interaction between modalities. Therefore, they proposed Multimodal VoxelNet (MVX-Net), a simple method that fuses image and LiDAR cloud point early and leverages VoxelNet [36] architecture to extend it. They presented two architectures: (i) PointFusion (PF) projects the point clouds onto images and extracts image features using a pre-trained 2D CNN detector. Next, the concatenated image features are fused with the corresponding voxelized point cloud point-wisely and fed to VFE feature encoding, followed by a 3D region proposal network (RPN) to generate the 3D bounding boxes. Fig. 40 illustrates the MVX-Net PointFusion architecture. (ii) VoxelFusion (VF) uses VoxelNet to create 3D voxels from the point cloud, project voxels to images, and extract image features for each voxel using the same pre-trained 2D detector. Next, the features are appended through VFE feature encoding, followed by 3D RPN to generate 3D bounding boxes. Fig. 41 illustrates

Fig. 44. 3D-CVF network architecture [62].

Fig. 45. CLOCs network architecture [63].

the MVX-Net VoxelFusion learning procedure. The empirical result showed that the MVX-Net PF has better performance over the KITTI dataset; however, VF has better performance than previous models.

LaserNet. Meyer et al. [60] proposed a new approach to detecting 3D objects; instead of predicting the 3D bounding box deterministically, they modelled each detection as a distribution. Therefore LaserNet predicted a multimodal distribution for each point and, by fusing them, predicted the probability of objects' 3D bounding boxes corners and the class of objects. In this way, the driving system behaviour can be adjusted based on the probability of predictions and the level of uncertainty. This method showed better detection performance and significant running time reduction due to the usage of dense range images instead of sparse BEV images. As Fig. 42 shows, the three different image

captures: Height, Intensity and Range, are fed to a fully convolutional network to generate a set of predictions: Box Parameters, Box Corner Variant and Class Probability. This set of predictions is provided per point. Then a probability distribution is calculated for each bounding box from BEV. A Mean Shifting Clustering combines and reduces the noise in each prediction. The training is end-to-end with a loss function defining 3D bounding box corners. An adaptive non-maximum suppression (NMS) algorithm eliminates duplicated distribution of boxes at the last module.

LaserNet++. Meyer et al. [61] extended their LaserNet 3D object detection by introducing LaserNet++. They added RGB image fusion as a new input to the network to enhance the detection performance, especially at long ranges. LaserNet gives a set of outputs as separate

Fig. 46. RVF-Net network architecture [64].

Fig. 47. CenterFusion network architecture [65].

tasks. However, the LaserNet++ combines 3D object detection and semantic segmentation in a single model to enhance the computational performance limited on self-driving vehicles. Another advantage of this model is that it does not require labelled images. Moreover, by adding unlabelled images, the model provides semantic segmentation with low runtime and better detection accuracy. Fig. 43 shows the proposed model network architecture.

3D-CVF. Fusing camera and LiDAR sensor signals improve the accuracy and robustness of 3D object detection due to their signals' unique characteristics and distribution. Using cameras and LiDAR together, a significant difference in the derived spatial feature maps is shown in both world and camera coordinates; this presents a significant challenge to the fusion process; Thus, fusing two heterogeneous feature maps involves a lot of effort to avoid information loss. Adopting the cross-view spatial feature fusion strategy, Yoo et al. [62] proposed a method called 3D-CVF to address this problem. In the method's initial step, Auto-calibrated projection is used to transform the 2D camera features into a smooth spatial feature map that corresponds most closely to the LiDAR features in the BEV. In the subsequent proposal refinement stage, both camera and LiDAR features are merged based on the spatial attention maps based on the camera location. Moreover, camera-LiDAR features are also fused from the subsequent proposal refinement stage. To improve proposal refinement, the low-level LiDAR features and camera features are supplemented with the joint camera-LiDAR features by using a region of interest (RoI)-based feature pooling method. They report that the proposed 3D-CVF achieves state-of-the-art performance in KITTI and nuScenes 3D object detection datasets and is significantly faster than LiDAR-only methods. As Fig. 44 presents, Following the division into backbone networks, each backbone network processes the camera-view images separately. Each backbone network then uses an automatically calibrated feature projection to transform the camera-view features into BEV features. Next, the camera and LiDAR data are fused using a gated feature fusion network. The detection outputs are predicted using a 3D ROI-based fusion network.

CLOCs. Pang et al. [63] proposed a fast multimodal fusion object detection framework with a low memory footprint and complexity. Camera-LiDAR Object Candidates (CLOCs) model generates 2D and 3D detection candidates based on 2D RGB images and LiDAR point cloud, then it encodes them into a set of joint candidates. Next, a 2D convolutional network process the non-empty candidates and then the result tensor is mapped to the probability of objects at the output. This method is fast due to the sparsity of the input and uses pre-trained 2D and 3D detectors without training them.

Fig. 45

RVF-Net. Nobis et al. [64] proposed a scheme for early sensor fusion. Lidar points were coloured with RGB information from the cameras. The radar points, their radar cross-section (RCS), and velocity features of these coloured lidar points are then fused with those of the radar. Using voxel-based processing, the network outputs predicted bounding boxes based on the points in the voxel structure. RadarVoxelFusionNet (RVF-Net) was presented as the most reliable parameterization after a detailed analysis of several parameterizations. A fusion network was also developed with radar, lidar, and camera data to detect 3D objects.

Fig. 48. SLS-Fusion pipeline architecture.

Fig. 49. Cross-Modality Object Detection [69].

Networks outperform lidar for late fusion as well as a Kalman Filter. In addition, they developed an alternative loss function that replaces simple discontinuous yaw parameterization. Fig. 46 presents the RVF-Net network architecture.

CenterFusion. Nabati et al. [65] proposed a middle-fusion approach that uses RGB images and a LiDAR point cloud. The main idea of Center-Fusion is to use a network to detect the centre point of objects in the images and then use a frustum-based method to map the LiDAR point cloud to the detected centre points. Finally, the associated points to a centre point generate feature maps and predict object properties. In particular, as Fig. 47 shows, initial 3D boxes are extracted from the image using image features, then a frustum module maps the boxes to an associated object detected from LiDAR to generate a LiDAR feature map. Both features maps are concatenated to regress and refine the object's depth, velocity, rotation and 3d bounding box. This framework is tested over the nuScenes dataset, which showed a 12% improvement in the nuScenes Detection Score (NDS) [66].

Fig. 50. Deep continuous fusion architecture [72].

Fig. 51. Proposed pipeline [74].

Fig. 52. Pseudo-LiDAR++ pipeline architecture [68].

SLS-Fusion. LiDAR sensors can provide very accurate depth information, but this information is very sparse and uneven due to the type of object and cause issues to detect the obstacles. This issue gets worth when LiDAR usually generates fewer points for small objects like pedestrians and cycles. The advantage of pseudo-LiDAR is that it uses images to generate points, especially enough for small objects and combines depth estimation to improve detection performance. To address mentioned points, Mai et al. [67] proposed a novel approach that used stereo 4-beam LiDAR and stereo image for estimating the depth more precisely to enhance the other object detection models' performance. This method is considered low-cost because the 4-beam LiDAR is significantly cheaper than 32 or 64 beams LiDAR. They proposed a deep neural network to fuse stereo images and stereo LiDAR point cloud and outperform all low-cost methods. As Fig. 48 shows, the stereo image is fed to a weight-sharing ResNet to encode features. The same procedure is done for the sparse LiDAR point cloud. The corresponding image and LiDAR pairs are fused during the decoding stage, and the left and right features tensor is generated. Both left and right features are passed through Disparity Cost Volume (DeCV) [68] to extract the depth information.

Cross-modality object detection. LiDAR sensors provide geometry

Fig. 53. End-to-End Pseudo-LiDAR architecture [76].

information, and images include texture and semantic information. Encoding point clouds and projecting them on images causes information loss. Moreover, redundant information is processed by fusing multisensor features, lowering the computational performance. Zhu et al. [69] address these issues by proposing the Cross-Modality, a two-stage framework: region proposal stage and box refinement stage. The first stage is responsible for generating proposals by keeping the point cloud data structure and using PointNet++ [42] to learn directly from point clouds to classify and segment, and applying point-wise feature fusion and 2D and 3D bounding box coupling anchors. At this stage, they used Resnet-50 [70] and FPN [71] for learning feature maps. The second stage is responsible for RoI-wise feature fusion from the image and LiDAR feature maps and learning more robust features to predict confidence and bounding boxes. They also proposed using pseudo-LiDAR to enhance the detection accuracy for distant and small objects with few points in the point cloud. The empirical test over KITTI showed that this method helped to generate more robust representations. Fig. 49 illustrates the mentioned process.

Sequential fusion methods

In the category, one sensor output is fed into the pipeline and later, another sensor output is added to fuse with the learned feature maps. It means there is only one pipeline to carry on the processes.

UberATG-ContFuse. Liang et al. [72] proposed an end-to-end continuous fusion learner capable of detecting objects at different levels of resolution. This method uses RGB images as the input and performs different resolution levels of segmentation using ResNet. Then it continuously adds the feature maps to the associated resolution level in the LiDAR stream at Fusion Layers to detect the 3D objects in BEV. This method utilizes a Deep Parametric Continuous Convolution [73], a simple learnable operator to extend grid-structured convolution to non-grid convolutions. Moreover, the proposed continuous fusion layer overcomes the sparsity issue in camera view observation by handling spatially discrete features.

Fig. 50

Pseudo-LiDAR. Wang et al. [74] proposed a novel method to encode stereo image-based depth maps to pseudo-LiDAR presentation and use existing LiDAR-based object detection techniques. In this way, having an expensive LiDAR system is no longer essential. They argue that the main reason for image-based methods is not their quality but the poor representation way of depth in a ConvNet-based 3D object detection system. They proved that this approach increased detection accuracy from 22% to 74% over the KITTI dataset.

LiDAR output is in the form of either point cloud or BEV, in which the shapes and sizes of the body are constant concerning depth. In contrast, image-based depth is densely estimated for each pixel and is often represented as additional image channels, making distant objects smaller and challenging to detect. Worse, the pixel neighbourhoods in this show group parts of remote areas of three-dimensional space. This leads to object localization issues because convolutional networks rely on 2D convulsions in these channels. To address this issue, they proposed a two-step approach. First, the estimated depth map from the stereo/ monocular images is encoded into a 3D point cloud called pseudo-LiDAR. Second, they used AVOD [52] and Frustum PointNets [54] pipelines to train the model. Fig. 51 shows the proposed pipeline.

Pseudo-LiDAR++. The primary weakness of Pseudo-LiDAR is the depth estimation accuracy for faraway objects. More specifically, the 3D object detection error is systematic and stems from the error in depth estimation. The detected objects are reliable, but depth estimations are too close or far from the ground truth box. Therefore, a de-bias in estimating depth is needed to locate the estimations inside the ground truth boxes. This bias results from computing the depth indirectly from the disparity of left and right images and inverting it to compute depth. Despite using the deep neural network to improve this estimation, the error in the close objects is in order of cm but for distant objects is in order of meters. Therefore, You et al. [68] proposed Pseudo-LiDAR++ to fill this gap. To solve this bias, they proposed to adapt the estimated depth information with direct depth estimation coming from a very cheap 4 beams LiDAR (instead of 32 or 64 beams) that its output is very sparse and is not good enough to be used alone. The pair of stereo images and 4 beams of LiDAR

Fig. 54. PointPainting data preparation [77].

Table 4 3D object detection AP – detection rate (in%) on KITTI.

Input	Category	Methods	Speed (FPS)	Cars			Pedestrians			Cycles
				Easy	Mod	Hard	Easy	Mod	Hard	Easy	Mod	Hard
Images	Monocular based Methods	Deep MANTA	NA	97.6	90.9	82.7	–	–	–	–	–	–
		Deep3DBox	–	–	–	–	–	–	–	–	–	–
		Mono3D	33.3	94.6	91.6	84.9	58.3	46.7	41.9	70.8	54.8	47.6
		GS3D	0.5	86.2	76.4	62.7	–	–	–	–	–	–
		Mono3D-PLiDAR	10.0	80.9	53.4	44.8	–	–	–	–	–	–
	Stereo-based Methods	3DOP	0.8	–	–	–	–	–	–	–	–	–
		Stereo R-CNN	3.3	94.0	86.0	71.3	–	–	–	–	–	–
Point Cloud	Multi-view-based Methods	VeloFCN	1.0	–	–	–	–	–	–	–	–	–
		PIXOR	28.6	–	–	–	–	–	–	–	–	–
	Voxel-based Methods	HVNet	31.0	–	–	–	–	–	–	–	–	–
		VoxelNet	4.3	–	–	–	–	–	–	–	–	–
		SECOND	20.0	96.4	95.8	90.6	–	–	–	–	–	–
		PointPillars	62.0	94.0	91.2	88.2	65.3	55.1	52.4	84.0	69.0	62.2
		Part-A2	12.5	95.0	91.9	89.1	68.8	58.0	54.0	88.9	78.3	71.2
	Point-based Methods	PointRCNN	10.0	95.9	91.9	87.1	58.9	49.4	46.4	86.2	73.4	66.5
		Point-GNN	1.7	96.6	93.5	88.4	71.6	58.2	54.1	85.8	75.1	68.7
		3DSSD	25.0	97.7	95.1	92.2	72.3	60.5	56.3	87.1	74.1	67.7
	Point-Voxel Methods	Fast Point R-CNN	16.7	96.1	93.2	87.7	–	–	–	–	–	–
		SA-SSD	25.0	97.9	95.2	90.2	–	–	–	–	–	–
		STD	12.5	96.1	93.2	90.5	68.3	55.0	50.9	84.0	71.6	64.9
		PV-RCNN	12.5	98.2	94.7	92.0	68.9	58.4	55.4	86.6	80.4	73.6
Fusion	Parallel Methods	MV3D	2.8	–	–	–	–	–	–	–	–	–
		AVOD	12.5	95.2	89.9	82.8	50.9	39.4	35.8	66.5	52.6	46.4
		Frustum PointNet	2.1	95.9	95.2	85.4	89.8	80.1	75.1	86.9	73.2	65.2
		Multi-Level Fusion	–	90.4	87.3	76.8	–	–	–	–	–	–
		Frustum ConvNet	2.1	95.9	92.2	80.1	83.6	72.9	67.2	86.8	78.1	68.1
		MVF	–	–	–	–	–	–	–	–	–	–
		UberATG-MMF	12.5	97.4	94.3	89.9	–	–	–	–	–	–
		MVX-Net	–	–	–	–	–	–	–	–	–	–
		LaserNet	3.3	–	–	–	–	–	–	–	–	–
		LaserNet++	2.6	–	–	–	–	–	–	–	–	–
		3D-CVF	16.7	96.8	93.4	86.1	–	–	–	–	–	–
		CLOCs	10.0	96.8	96.1	91.1	–	–	–	–	–	–
		RVF-Net	22.7	–	–	–	–	–	–	–	–	–
		CenterFusion	–	–	–	–	–	–	–	–	–	–
		SLS-Fusion	–	–	–	–	–	–	–	–	–	–
		Cross-Modality	–	96.2	93.5	88.7	–	–	–	–	–	–
	Sequential Methods	UberATG-ContFuse	16.7	–	–	–	–	–	–	–	–	–
		Pseudo-LiDAR	2.5	85.4	67.8	58.5	–	–	–	–	–	–
		Pseudo-LiDAR++	2.5	94.5	82.9	75.5	–	–	–	–	–	–
		E2E Pseudo-LiDAR	–	–	–	–	–	–	–	–	–	–
		PointPainting	2.5	98.4	92.6	89.7	61.9	53.8	50.6	87.7	78.0	69.3

is ideal for de-bias the depth estimation using graph-based manifold learning [75]. Moreover, instead of estimating disparity from the stereo images, they estimated a depth grid. The estimated depth grid pixel from the stereo image is matched with a few corresponding point clouds with high depth accuracy by the nearest neighbour graph method. Then, the neighbour nodes are adjusted. Finally, they propagate this accurate depth information through the graph. Fig. 52 shows the mentioned procedure.

End-to-End Pseudo-LiDAR. End-to-End Pseudo-LiDAR is an improvement of Pseudo-LiDAR [74] proposed by the same research team. In Pseudo-LiDAR, the depth information is calculated by the disparity between the left and right image and the extracted depth information is used in the object detection network. In particular, two separate neural networks are trained: a depth estimator trained on stereo images and an object detector trained on pseudo-LiDAR point cloud; combining these two networks may result in an unpleasant performance. Therefore, the main issue is that there are two training objectives, and their optimization is not necessarily aligned to the detection performance. Qian et al. [64] proposed a framework based on differentiable Change of Representation (CoR) to perform end-to-end training to address this issue. CoR differentiates between the depth estimator and object detector, enabling end-to-end backpropagation, which uses object detection loss to train the whole pipeline. There are two types of CoS: subsampling and quantization. To prevail over the non-differentiability of CoR, they introduced a differentiable soft quantization. The most important advantage of this method is that it is compatible with different LiDAR-based object detection like PIXOR [34], and PointRCNN [40]. Fig. 53 shows the forward propagation (black) and backpropagation (blue) through the entire pipeline.

PointPainting. The LiDAR-only methods outperform fusion-based methods, while images add complementary information to the object detector. Therefore, Vora et al. [66] filled mentioned gap by introducing PointPainting. The main idea is to paint each LiDAR point with the output of the image semantic segmentation class. Then the result can be fed to every LiDAR-only method. The experiment over the KITTI dataset showed a significant improvement in object detection using PointRCNN [40], VoxelNet [36] and PointPillar [38]. In particular, each LiDAR point is projected to image semantic segmentation, and a new activation channel is concatenated to the points as a bundle. Then this bundle is fed to any LiDAR only 3D object detection as mentioned. Fig. 54 illustrates the procedure.

Discussion

We can see that most methods use the KITTI test dataset to train and evaluate their performance within Cars, Pedestrians and Cycles in 3 hardness classes: easy, moderate and hard. However, more recent datasets have more samples generated under different weather and light

Table 5 3D object detection AP - object orientation (in%) on KITTI.

Input	Category	Methods	Speed (FPS)	Cars			Pedestrians			Cycles
				Easy	Mod	Hard	Easy	Mod	Hard	Easy	Mod	Hard
Images	Monocular based Methods	Deep MANTA	–	97.44	61.49	52.58	–	–	–	–	–	–
		Deep3DBox	–	92.9	88.75	76.76	–	–	–	–	–	–
		Mono3D	33.33	93.54	90.02	83.19	47.46	36.95	32.87	56.78	42.29	36.54
		GS3D	0.50	85.79	75.63	61.85	–	–	–	–	–	–
		Mono3D-PLiDAR	10.00	76.9	49.39	41.13	–	–	–	–	–	–
	Stereo-based Methods	3DOP	0.83	91.44	86.1	76.52	72.94	59.8	57.03	70.13	58.68	52.35
		Stereo R-CNN	3.33	–	–	–	–	–	–	–	–	–
Point Cloud	Multi-view-based Methods	VeloFCN	1.00	59.1	45.9	41.1	–	–	–	–	–	–
		PIXOR	28.60	–	–	–	–	–	–	–	–	–
	Voxel-based Methods	HVNet	31.00	–	–	–	–	–	–	–	–	–
		VoxelNet	4.35	–	–	–	–	–	–	–	–	–
		SECOND	20.00	96.42	95.67	90.37	–	–	–	–	–	–
		PointPillars	62.00	93.84	90.7	87.47	57.47	48.05	45.4	83.79	68.55	61.71
		Part-A2	12.50	95	91.73	88.86	63.51	52.2	48.27	88.7	77.52	70.41
	Point-based Methods	PointRCNN	10.00	95.9	91.77	86.92	57.19	47.33	44.31	85.94	72.81	65.84
		Point-GNN	1.67	38.66	37.2	36.29	39.16	31.86	29.65	36.29	32.37	29.81
		3DSSD	25.00	–	–	–	–	–	–	–	–	–
	Point-Voxel Methods	Fast Point R-CNN	16.67	–	–	–	–	–	–	–	–	–
		SA-SSD	25.00	39.4	38.3	37.07	–	–	–	–	–	–
		STD	12.50	–	–	–	–	–	–	–	–	–
		PV-RCNN	12.50	98.15	94.57	91.85	63.45	52.42	49.23	86.43	79.7	72.96
Fusion	Parallel Methods	MV3D	2.78	–	–	–	–	–	–	–	–	–
		AVOD	12.50	94.98	89.22	82.14	42.54	32.19	29.09	64.81	51.05	45.12
		Frustum PointNet	2.13	–	–	–	–	–	–	–	–	–
		Multi-Level Fusion	–	90.35	87.03	76.37	–	–	–	–	–	–
		Frustum ConvNet	2.13	95.81	91.98	79.83	75.19	63.87	58.57	86.39	76.71	66.92
		MVF	–	–	–	–	–	–	–	–	–	–
		UberATG-MMF	12.50	–	–	–	–	–	–	–	–	–
		MVX-Net	–	–	–	–	–	–	–	–	–	–
		LaserNet	3.33	–	–	–	–	–	–	–	–	–
		LaserNet++	2.63	–	–	–	–	–	–	–	–	–
		3D-CVF	16.67	40.44	39.79	36.1	–	–	–	–	–	–
		CLOCs	10.00	96.77	95.93	90.93	–	–	–	–	–	–
		RVF-Net	22.7	–	–	–	–	–	–	–	–	–
		CenterFusion	–	–	–	–	–	–	–	–	–	–
		SLS-Fusion	–	–	–	–	–	–	–	–	–	–
		Cross-Modality	–	–	–	–	–	–	–	–	–	–
	Sequential Methods	UberATG-ContFuse	16.67	–	–	–	–	–	–	–	–	–
		Pseudo-LiDAR	2.50	–	–	–	–	–	–	–	–	–
		Pseudo-LiDAR++	2.50	94.14	81.87	74.29	–	–	–	–	–	–
		E2E Pseudo-LiDAR	–	–	–	–	–	–	–	–	–	–

		PointPainting	2.50	98.36	92.43	89.49	59.25	50.22	46.95	87.33	76.92	68.21

conditions. The summary and empirical results are shown in Table 3 -7. It is not easy to name a model as the best 3D object detector because some models detect cars, pedestrians and cycles. In contrast, some only detect cars, some models predict the bounding box orientation, but some do not. Furthermore, some have a reasonable detection rate or are faster than others. Some operate only if an expensive LiDAR sensor has been used; some do not. Considering all these different factors does not lead us to select the best model. Instead, we can infer that one or multiple models can be utilized to give a satisfying performance based on the application. More importantly, it means that 3D object detection models are still immature, and there is a significant need for future research to optimize all these factors simultaneously.

A comparison of the existing 3D object detection methods is illustrated in Table 3. We have provided a classification of the literature into three main categories, including 1) Monocular/ Stereo Image-based methods, 2) Point cloud-based methods, and 3) Fusion-based methods. We have also provided another level of categorization for each class. We differentiated each method based on category, subcategory, modality, number of stages, and datasets used. We can see that most of the methods use the KITTI test dataset to train and evaluate their performance within Cars, Pedestrians and Cycles in 3 hardness classes: easy, moderate and hard. However, more recent datasets have more samples generated under different weather and light conditions. In self-driving cars, to recognize surrounding objects in three dimensions, we recognize the essential factors in evaluating the performance of a model as:

Object detection rate, the total correct object detections out of the total object in the scenes (Table 4)
Detection speed in revelling surrounding objects in real-time (Table 6, Speed (FPS) column)
Object orientation detection the accuracy of bounding boxes; orientation (Table 5)
Accuracy of the bounding boxes around objects from the front perspective (Table 6)
Accuracy of the bounding boxes around objects from a BEV perspective (Table 7)

As we can see in Tables 4-7, some of the factors are not available due to the scop of the model operation, the properties of the model's outputs and the dataset that the model is developed based on rather than KITTI, which is the most commonly used dataset to compare the models' performance. Moreover, from Table 4, some models are developed only to

Table 6 3D object detection AP - 3D (in%) on KITTI.

Input	Category	Methods	Speed (FPS)	Cars			Pedestrians			Cycles
				Easy	Mod	Hard	Easy	Mod	Hard	Easy	Mod	Hard
Images	Monocular based Methods	Deep MANTA	–	–	–	–	–	–	–	–	–	–
		Deep3DBox	–	93.0	89.0	77.2	–	–	–	–	–	–
		Mono3D	33.3	23.4	15.3	12.8	10.1	6.6	5.5	4.7	2.5	2.2
		GS3D	0.5	4.5	2.9	2.5	–	–	–	–	–	–
		Mono3D-PLiDAR	10.0	10.8	7.5	6.1	–	–	–	–	–	–
	Stereo-based Methods	3DOP	0.8	93.0	88.6	79.1	81.8	67.5	64.7	78.4	68.9	61.4
		Stereo R-CNN	3.3	47.6	30.2	23.7	–	–	–	–	–	–
Point Cloud	Multi-view-based Methods	VeloFCN	1.0	60.3	47.5	42.7	–	–	–	–	–	–
		PIXOR	28.6	81.7	77.1	73.0	–	–	–	–	–	–
	Voxel-based Methods	HVNet	31.0	–	–	–	–	–	–	–	–	–
		VoxelNet	4.3	82.0	65.5	62.9	57.9	53.4	48.9	67.2	47.7	45.1
		SECOND	20.0	87.4	79.5	74.0	51.1	42.6	37.3	70.5	53.9	46.9
		PointPillars	62.0	82.6	74.3	69.0	51.5	41.9	38.9	77.1	58.7	51.9
		Part-A2	12.5	87.8	78.5	73.5	53.1	43.4	40.1	79.2	63.5	56.9
	Point-based Methods	PointRCNN	10.0	87.0	75.6	70.7	48.0	39.4	36.0	75.0	58.8	52.5
		Point-GNN	1.7	88.3	79.5	72.3	51.9	43.8	40.1	78.6	63.5	57.1
		3DSSD	25.0	88.4	79.6	74.6	54.6	44.3	40.2	82.5	64.1	56.9
	Point-Voxel Methods	Fast Point R-CNN	16.7	85.3	77.4	70.2	–	–	–	–	–	–
		SA-SSD	25.0	88.8	79.8	74.2	–	–	–	–	–	–
		STD	12.5	88.0	79.7	75.1	53.3	42.5	38.4	78.7	61.6	55.3
		PV-RCNN	12.5	90.3	81.4	76.8	52.2	43.3	40.3	78.6	63.7	57.7
Fusion	Parallel Methods	MV3D	2.8	75.0	63.6	54.0	–	–	–	–	–	–
		AVOD	12.5	76.4	66.5	60.2	36.1	27.9	25.8	57.2	42.1	38.3
		Frustum PointNet	2.1	82.2	69.8	60.6	50.5	42.2	38.1	72.3	56.1	49.0
		Multi-Level Fusion	–	47.9	29.5	26.4	–	–	–	–	–	–

		Frustum ConvNet	2.1	87.4	76.4	66.7	52.2	43.4	38.8	82.0	65.1	56.5
		MVF	–	90.2	79.1	76.4	–	–	–	–	–	–
		UberATG-MMF	12.5	88.4	77.4	70.2	–	–	–	–	–	–
		MVX-Net	–	85.5	73.3	67.4	–	–	–	–	–	–
		LaserNet	3.3	–	–	–	–	–	–	–	–	–
		LaserNet++	2.6	–	–	–	–	–	–	–	–	–
		3D-CVF	16.7	89.2	80.1	73.1	–	–	–	–	–	–
		CLOCs	10.0	89.2	82.3	77.2	–	–	–	–	–	–
		RVF-Net	22.7	–	–	–	–	–	–	–	–	–
		CenterFusion	–	–	–	–	–	–	–	–	–	–
		SLS-Fusion	–	93.0	86.2	84.0	–	–	–	–	–	–
		Cross-Modality	–	87.2	77.3	72.0	–	–	–	–	–	–
	Sequential Methods	UberATG-ContFuse	16.7	83.7	68.8	61.7	–	–	–	–	–	–
		Pseudo-LiDAR	2.5	54.5	34.1	28.3	–	–	–	–	–	–
		Pseudo-LiDAR++	2.5	61.1	42.4	37.0	–	–	–	–	–	–
		E2E Pseudo-LiDAR	–	64.8	43.9	38.1	–	–	–	–	–	–
		PointPainting	2.5	82.1	71.7	67.1	50.3	41.0	37.9	77.6	63.8	55.9

detect cars, not pedestrians and cycles. The cost of implementing a model is another factor in evaluating a model.

Using LiDAR sensors imposes a higher cost on cars; However, the information received from LiDAR helps us determine the depth accurately. Moreover, models that learn directly from the point cloud show less computational complexity. Therefore, there should always be a trade-off between model accuracy and implementation cost. Considering these two significant factors, Pseudo LiDAR-based models tried to make a good balance between these two factors. However, the FPS (2.5) and the scope of the operation are not well satisfying.

In summary, the best model in object detection is CLOCs, SA-SSD and SECOND, and the worst are Mono3D-PLiDAR, AVOD and Mono3D. Regarding 3D box orientation performance, the best models are CLOCs, SECOND, and the worst are Point-GNN and SA-SSD. The best AP-3D performance belongs to SLS-Fusion and Deep3Dbox, and the worst is GS3D. SA-SSD and SLS-Fusion have the best performance regarding the BEV performance, and GS3D has the worst. Finally, PointPillar has the best performance of 62 FPS regarding the detection per second rate.

Conclusion

In this paper, we provided a comprehensive overview of the state-of-

the-art 3D object detection methods. Many different 3D object detection methods have been proposed to tackle different scenarios during autonomous driving. We reviewed the architecture and limitation of a monocular image, the different representations of point clouds, the fusion of modalities, the cons and pros of each approach, and the computational complexity that decides which method could be beneficial in real-time application. Despite all efforts, there is still a significant need for a cheap, accurate and real-time model. Moreover, we reviewed the occultation, and truncation issues when using the image and different methods to extract depth information from an image. Despite many efforts to determine the depth of the small and distant objects, there is a research gap to enhance the accuracy of models. In addition, the fusion of the inputs is an ample opportunity for future research since a variety of fusion techniques are possible to be conducted. Future directions include the investigations of federated learning object detection-based methods in real-time for large-sale distributed datasets on the edge and the cloud.

Declaration of Competing Interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Table 7 3D object detection AP - BEV (in%) on KITTI.

Input	Category	Methods	Speed (FPS)	Cars			Pedestrians			Cycles
				Easy	Mod	Hard	Easy	Mod	Hard	Easy	Mod	Hard
Images	Monocular based Methods	Deep MANTA	–	–	–	–	–	–	–	–	–	–
		Deep3DBox	–	–	–	–	–	–	–	–	–	–
		Mono3D	33.3	31.6	20.8	17.7	11.9	8.0	6.8	5.8	3.3	2.7
		GS3D	0.5	8.4	6.1	4.9	–	–	–	–	–	–
		Mono3D-PLiDAR	10.0	21.3	13.9	11.3	–	–	–	–	–	–
	Stereo-based Methods	3DOP	0.8	–	–	–	–	–	–	–	–	–
		Stereo R-CNN	3.3	61.9	41.3	33.4	–	–	–	–	–	–
Point Cloud	Multi-view-based Methods	VeloFCN	1.0	–	–	–	–	–	–	–	–	–
		PIXOR	28.6	–	–	–	–	–	–	–	–	–
	Voxel-based Methods	HVNet	31.0	92.8	88.8	83.4	54.8	48.9	46.3	84.0	71.2	63.7
		VoxelNet	4.3	89.6	84.8	78.6	66.0	61.1	57.0	74.4	52.2	50.5
		SECOND	20.0	88.1	79.4	78.0	55.1	46.3	44.8	73.7	56.0	48.8
		PointPillars	62.0	90.1	86.6	82.8	57.6	48.6	45.8	79.9	62.7	55.6
		Part-A2	12.5	91.7	87.8	84.6	59.0	49.8	45.9	83.4	68.7	61.9
	Point-based Methods	PointRCNN	10.0	92.1	87.4	82.7	54.8	46.1	42.8	82.6	67.2	60.3
		Point-GNN	1.7	93.1	89.2	83.9	55.4	47.1	44.6	81.2	67.3	59.7
		3DSSD	25.0	92.7	89.0	85.9	60.5	49.9	45.7	85.0	67.6	61.1
	Point-voxel Methods	Fast Point R-CNN	16.7	90.9	87.8	80.5	–	–	–	–	–	–
		SA-SSD	25.0	95.0	91.0	86.0	–	–	–	–	–	–
		STD	12.5	94.7	89.2	86.4	60.0	48.7	44.6	81.4	67.2	59.4
		PV-RCNN	12.5	95.0	90.7	86.1	59.9	50.6	46.7	82.5	68.9	62.4
Fusion	Parallel Methods	MV3D	2.8	86.6	78.9	69.8	–	–	–	–	–	–
		AVOD	12.5	89.8	85.0	78.3	42.6	33.6	30.1	64.1	48.2	42.4
		Frustum PointNet	2.1	91.2	84.7	74.8	57.1	49.6	45.5	77.3	61.4	53.8
		Multi-Level Fusion	–	55.0	36.7	31.5	–	–	–	–	–	–
		Frustum ConvNet	2.1	91.5	85.8	76.1	57.0	49.0	44.3	84.2	68.9	60.1
		MVF	–	–	–	–	–	–	–	–	–	–
		UberATG-MMF	12.5	93.7	88.2	82.0	–	–	–	–	–	–
		MVX-Net	–	89.5	84.9	79.0	–	–	–	–	–	–
		LaserNet	3.3	78.3	73.8	66.5	–	–	–	–	–	–
		LaserNet++	2.6	86.2	0.0	0.0	65.7	0.0	0.0	83.4	0.0	0.0
		3D-CVF	16.7	3.5	89.6	82.5	–	–	–	–	–	–
		CLOCs	10.0	92.9	89.5	86.4	–	–	–	–	–	–
		RVF-Net	22.7	–	–	–	–	–	–	–	–	–
		CenterFusion	–	–	–	–	–	–	–	–	–	–
		SLS-Fusion	–	93.2	88.8	83.4	–	–	–	–	–	–
		Cross-Modality	–	–	–	–	–	–	–	–	–	–
	Sequential Methods	UberATG-ContFuse	16.7	94.1	85.4	75.9	–	–	–	–	–	–
		Pseudo-LiDAR	2.5	67.3	45.0	38.4	–	–	–	–	–	–
		Pseudo-LiDAR++	2.5	78.3	58.0	51.3	–	–	–	–	–	–
		E2E Pseudo-LiDAR	–	79.6	58.8	52.1	–	–	–	–	–	–
		PointPainting	2.5	92.5	88.1	83.4	58.7	49.9	46.3	83.9	71.5	63.0

References

[1] SAE on-road Automated Vehicle Standards Committee, J3016 taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles, SAE Int. (2018). [https://www.sae.](https://www.sae.org/standards/content/j3016_201806/(accessed)
org/standards/content/j3016\_201806/(accessed. [2] Synopsys, The 6 levels of vehicle autonomy explained | synopsys automotive, Synpsys.Com (2019). https://www.synopsys.com/automotive/autonomous driving-levels.html. accessed Nov. 03, 2021.
[3] F. Rosique, P.J. Navarro, C. Fernandez, ´ A. Padilla, A systematic review of perception system and simulators for autonomous vehicles research, Sensors (Switzerland) 19 (3) (Feb. 2019), https://doi.org/10.3390/s19030648.
[4] B. Ranft, C. Stiller, The role of machine vision for intelligent vehicles, IEEE Trans. Intell. Veh. 1 (1) (2016) 8–19, https://doi.org/10.1109/TIV.2016.2551553.
[5] M. Weber, P. Wolf, J.M. Zollner, DeepTLR: a single deep convolutional network for detection and classification of traffic lights, in: IEEE Intelligent Vehicles Symposium, Proceedings, Aug. 2016, pp. 342–348, https://doi.org/10.1109/ IVS.2016.7535408, 2016-Augus.
[6] Sony Corporation, "Sony commercializes the industry's first*1 high-sensitivity CMOS image sensor for automotive cameras, delivering simultaneous LED flicker mitigation and high-quality HDR shooting, News Rel. (2017). https://www.sony. com/en\_us/SCA/company-news/press-releases/sony-electronics/2017/sony-c ommercializes-the-industrys-first1-automotive-2mp-highsensitivity-cmos-
image-sensor-with-led-flicker-mitigation-and-hdr.html. accessed Oct. 07, 2021. [7] S. Sivaraman, M.M. Trivedi, Looking at vehicles on the road: a survey of visionbased vehicle detection, tracking, and behavior analysis, IEEE Trans. Intell. Transp. Syst. 14 (4) (Dec. 2013) 1773–1795, https://doi.org/10.1109/TITS.2013.2266661.
[8] "Velodyne HDL-64E | AutonomouStuff," 2018. https://autonomoustuff. com/products/velodyne-hdl-64e (accessed Oct. 07, 2021).
[9] M. Hall, "Velodyne LiDAR launches VLS-128, the world's highest resolution LiDAR for autonomous vehicles." https://www.businesswire.com/news/home/ 20171107005794/en/Velodyne-LiDAR-Launches-VLS-128™-World's-Highest-Resolution (accessed Oct. 07, 2021).
[10] A. Geiger, P. Lenz, C. Stiller, R. Urtasun, The KITTI vision benchmark suite, KITTI Vis. Benchmark Suite (2013) 1–13. Accessed: Oct. 07, 2021. [Online]. Available, http://www.cvlibs.net/datasets/kitti/.
[11] A. Geiger, P. Lenz, R. Urtasun, Are we ready for autonomous driving? the KITTI vision benchmark suite, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2012, pp. 3354–3361, https://doi. org/10.1109/CVPR.2012.6248074.
[12] H. Caesar, et al., Nuscenes: a multimodal dataset for autonomous driving, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 11618–11628, https://doi.org/10.1109/ CVPR42600.2020.01164.
[13] P. Sun, et al., Scalability in perception for autonomous driving: waymo open dataset, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 2443–2451, https://doi.org/10.1109/ CVPR42600.2020.00252.
[14] J. Houston et al., "One thousand and one hours: self-driving motion prediction dataset," Jun. 2020, Accessed: Oct. 14, 2021. [Online]. Available: http://arxiv. org/abs/2006.14480.
[15] A. Patil, S. Malla, H. Gang, Y.T. Chen, The H3D dataset for full-surround 3D multiobject detection and tracking in crowded urban scenes, in: Proceedings - IEEE International Conference on Robotics and Automation, Mar. 2019, pp. 9552–9557, https://doi.org/10.1109/ICRA.2019.8793925, 2019-May.
[16] Y. Ma, X. Zhu, S. Zhang, R. Yang, W. Wang, and D. Manocha, "Apollo Scape," Proceedings of the AAAI conference on artificial intelligence, 2019. http://apolloscape. auto/trajectory.html#to_download_href (accessed Oct. 14, 2021).
[17] M.F. Chang, et al., Argoverse: 3D tracking and forecasting with rich maps, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019, pp. 8740–8749, https://doi.org/10.1109/ CVPR.2019.00895, 2019-June.
[18] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, "CARLA: an open urban driving simulator," 2017, Accessed: Oct. 07, 2021. [Online]. Available: htt p://arxiv.org/abs/1711.03938.
[19] M. Müller, V. Casser, J. Lahoud, N. Smith, B. Ghanem, Sim4CV: a photo-realistic simulator for computer vision applications, Int. J. Comput. Vis. 126 (9) (2018) 902–919, https://doi.org/10.1007/s11263-018-1073-7.
[20] J. Hosang, R. Benenson, P. Dollar, B. Schiele, What makes for effective detection proposals? IEEE Trans. Pattern Anal. Mach. Intell. 38 (4) (Apr. 2016) 814–830, https://doi.org/10.1109/TPAMI.2015.2465908.
[21] K.E.A. Van De Sande, J.R.R. Uijlings, T. Gevers, A.W.M. Smeulders, Segmentation as selective search for object recognition, in: Proc. IEEE Int. Conf. Comput. Vis, 2011, pp. 1879–1886,https://doi.org/10.1109/ICCV.2011.6126456.
[22] C.L. Zitnick, P. Doll´ ar, Edge boxes: locating object proposals from edges, Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformat.) 8693 (5) (2014) 391–405, https://doi.org/10.1007/978-3-319- 10602-1\_26. LNCSPART.
[23] S. Ren, K. He, R. Girshick, J. Sun, Faster R-CNN: towards real-time object detection with region proposal networks, IEEE Trans. Pattern Anal. Mach. Intell. 39 (6) (Jun. 2017) 1137–1149, https://doi.org/10.1109/TPAMI.2016.2577031.
[24] X. Chen, K. Kundu, Y. Zhu, H. Ma, S. Fidler, R. Urtasun, 3D object proposals using stereo imagery for accurate object class detection, IEEE Trans. Pattern Anal. Mach. Intell. 40 (5) (2018) 1259–1272, https://doi.org/10.1109/TPAMI.2017.2706685.
[25] P. Li, H. Zhao, Monocular 3D object detection using dual quadric for autonomous driving, Neurocomputing 441 (2021) 151–160, https://doi.org/10.1016/j. neucom.2021.01.110.
[26] F. Chabot, M. Chaouch, J. Rabarisoa, C. Teuli`ere, T. Chateau, Deep MANTA: a coarse-to-fine many-task network for joint 2D and 3D vehicle analysis from monocular image, in: Proc. - 30th IEEE Conf. Comput. Vis. Pattern Recognition, Mar. 2017, pp. 1827–1836. CVPR 2017, vol. 2017-JanuaAccessed: Oct. 02, 2021. [Online]. Available, https://arxiv.org/abs/1703.07570v1.
[27] Y. Xiang, W. Choi, Y. Lin, S. Savarese, Data-driven 3D Voxel Patterns for object category recognition, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015, pp. 1903–1911, https://doi.org/ 10.1109/CVPR.2015.7298800, 07-12-June.
[28] Y. Xiang, W. Choi, Y. Lin, S. Savarese, Subcategory-aware convolutional neural networks for object proposals and detection, in: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Mar. 2017, pp. 924–933, https://doi. org/10.1109/WACV.2017.108.
[29] A. Mousavian, D. Anguelov, J. Flynn, J. Kosecka, J. Koˇseck´ a, J. Flynn, 3D bounding box estimation using deep learning and geometry, in: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5632–5640, https://doi.org/10.1109/CVPR.2017.597. *CVPR*20172017-Janua.
[30] B. Li, W. Ouyang, L. Sheng, X. Zeng, X. Wang, GS3D: an efficient 3D object detection framework for autonomous driving, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2019) 1019–1028, https://doi.org/10.1109/ CVPR.2019.00111, 2019-June.
[31] P. Li, X. Chen, S. Shen, Stereo R-CNN based 3D object detection for autonomous driving, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2019) 7636–7644, https://doi.org/10.1109/CVPR.2019.00783, 2019-June.
[32] D. Deery, J. Jimenez-Berni, H. Jones, X. Sirault, R. Furbank, Proximal remote sensing buggies and potential applications for field-based phenotyping, Agronomy 4 (3) (2014) 349–379, https://doi.org/10.3390/agronomy4030349.
[33] B. Li, T. Zhang, T. Xia, Vehicle detection from 3D lidar using fully convolutional network, Robot. Sci. Syst. 12 (Aug. 2016), https://doi.org/10.15607/rss.2016.
xii.042. [34] B. Yang, W. Luo, R. Urtasun, PIXOR: real-time 3D object detection from point clouds, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018, pp. 7652–7660, https://doi.org/10.1109/ CVPR.2018.00798.
[35] G. Adusumilli, 3D object detection classifier in 6 steps using kaolin and colab | towards data science, Toward Data Sci. (2020). https://towardsdatascience. com/3d-object-classification-in-6-steps-using-kaolin-and-colab-9ecb079143a8. accessed Oct. 13, 2021.
[36] Y. Zhou, O. Tuzel, VoxelNet: End-to-End learning for point cloud based 3D object detection, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018, pp. 4490–4499, https://doi.org/10.1109/ CVPR.2018.00472.
[37] Y. Yan, Y. Mao, B. Li, Second: sparsely embedded convolutional detection, Sensors (Switzerland) 18 (10) (Oct. 2018), https://doi.org/10.3390/s18103337.
[38] A.H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, O. Beijbom, Pointpillars: fast encoders for object detection from point clouds, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019, pp. 12689–12697, https://doi.org/10.1109/CVPR.2019.01298, 2019-June.
[39] S. Shi, Z. Wang, J. Shi, X. Wang, H. Li, From points to parts: 3D object detection from point cloud with part-aware and part-aggregation network, IEEE Trans. Pattern Anal. Mach. Intell. 43 (8) (Jul. 2021) 2647–2664, https://doi.org/ 10.1109/TPAMI.2020.2977026.
[40] S. Shi, X. Wang, H. Li, PointRCNN: 3D object proposal generation and detection from point cloud, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019, pp. 770–779, https://doi.org/ 10.1109/CVPR.2019.00086, 2019-June.
[41] M. Ye, S. Xu, T. Cao, HVNet: hybrid voxel network for LiDAR based 3D object detection, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 1628–1637, https://doi.org/10.1109/ CVPR42600.2020.00170.
[42] C.R. Qi, L. Yi, H. Su, L.J. Guibas, PointNet++: deep hierarchical feature learning on point sets in a metric space, Adv. Neural Inf. Process. Syst. (Jun. 2017) 5100–5109, 2017-DecemAccessed: Oct. 14, 2021. [Online]. Available, https://arxiv.org/abs
[43] C.R. Qi, H. Su, K. Mo, L.J. Guibas, PointNet: deep learning on point sets for 3D classification and segmentation, in: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 77–85, https://doi.org/ 10.1109/CVPR.2017.16. *CVPR*20172017-Janua.
[44] Z. Yang, Y. Sun, S. Liu, J. Jia, 3DSSD: point-based 3d single stage object detector, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 11037–11045, https://doi.org/10.1109/ CVPR42600.2020.01105.
[45] W. Shi, R. Rajkumar, Point-GNN: graph neural network for 3D object detection in a point cloud, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 1708–1716, https://doi.org/10.1109/ CVPR42600.2020.00178.
[46] Victor Bapst, T. Keck, A. Grabska-Barwinska, C. Donner, Towards understanding glasses with graph neural networks, DeepMind (2020). https://deepmind.com/bl og/article/Towards-understanding-glasses-with-graph-neural-networks. accessed Nov. 03, 2021.
[47] Y. Chen, S. Liu, X. Shen, J. Jia, Fast point R-CNN, in: Proceedings of the IEEE International Conference on Computer Vision, Aug. 2019, pp. 9774–9783, https:// doi.org/10.1109/ICCV.2019.00987, 2019-Octob.
[48] Z. Yang, Y. Sun, S. Liu, X. Shen, J. Jia, STD: sparse-to-dense 3D object detector for point cloud, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 1951–1960, https://doi.org/10.1109/ICCV.2019.00204, 2019- Octob.
[49] S. Shi, et al., PV-RCNN: point-voxel feature set abstraction for 3D object detection, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 10526–10535, https://doi.org/10.1109/ CVPR42600.2020.01054.
[50] C. He, H. Zeng, J. Huang, X.S. Hua, L. Zhang, Structure aware single-stage 3D object detection from point cloud, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 11870–11879, https://doi.org/10.1109/CVPR42600.2020.01189.
[51] X. Chen, H. Ma, J. Wan, B. Li, T. Xia, Multi-view 3D object detection network for autonomous driving, in: Proc. - 30th IEEE Conf. Comput. Vis. Pattern Recognition, 2017, pp. 6526–6534, https://doi.org/10.1109/CVPR.2017.691. *CVPR*2017- Janua2017.
[52] J. Ku, M. Mozifian, J. Lee, A. Harakeh, S.L. Waslander, Joint 3D proposal generation and object detection from view aggregation, IEEE Int. Conf. Intell. Robot. Syst. (Dec. 2018) 5750–5757, https://doi.org/10.1109/ IROS.2018.8594049.
[53] X.T. Vo, T.D. Tran, D.L. Nguyen, K.H. Jo, Stair-step feature pyramid networks for object detection, Commun. Comput. Inf. Sci. 1405 (2021) 168–175, https://doi. org/10.1007/978-3-030-81638-4\_13.
[54] C.R. Qi, W. Liu, C. Wu, H. Su, L.J. Guibas, Frustum PointNets for 3D object detection from RGB-D data, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (Nov. 2018) 918–927, https://doi.org/10.1109/CVPR.2018.00102.
[55] B. Xu, Z. Chen, Multi-level fusion based 3D object detection from monocular images, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018, pp. 2345–2353, https://doi.org/10.1109/ CVPR.2018.00249.
[56] Z. Wang, K. Jia, Frustum ConvNet: sliding frustums to aggregate local point-wise features for amodal, IEEE Int. Conf. Intell. Robot. Syst. (Mar. 2019) 1742–1749, https://doi.org/10.1109/IROS40897.2019.8968513.
[57] Y. Zhou et al., "End-to-end multi-view fusion for 3D object detection in LiDAR point clouds," pp. 923–932, Oct. 2019, Accessed: Nov. 10, 2021. [Online]. Available: http://arxiv.org/abs/1910.06528.
[58] M. Liang, B. Yang, Y. Chen, R. Hu, R. Urtasun, Multi-task multi-sensor fusion for 3D object detection, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019, pp. 7337–7345, https://doi.org/ 10.1109/CVPR.2019.00752, 2019-June.
[59] V.A. Sindagi, Y. Zhou, O. Tuzel, MVX-net: multimodal VoxelNet for 3D object detection, in: Proceedings - IEEE International Conference on Robotics and Automation, May 2019, pp. 7276–7282, https://doi.org/10.1109/ ICRA.2019.8794195, 2019-May.
[60] G.P. Meyer, A. Laddha, E. Kee, C. Vallespi-Gonzalez, C.K. Wellington, Lasernet: an efficient probabilistic 3D object detector for autonomous driving, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2019) 12669–12678, https:// doi.org/10.1109/CVPR.2019.01296, 2019-June.
[61] G.P. Meyer, J. Charland, D. Hegde, A. Laddha, C. Vallespi-Gonzalez, Sensor fusion for joint 3D object detection and semantic segmentation, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 1230–1237, https://doi.org/10.1109/CVPRW.2019.00162, 2019-June.
[62] J.H. Yoo, Y. Kim, J. Kim, J.W. Choi, 3D-CVF: generating joint camera and LiDAR features using cross-view spatial feature fusion for 3D object detection, Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics) 12372 (Apr. 2020) 720–736, https://doi.org/10.1007/978-3-030-58583-9\_43. LNCS.
[63] S. Pang, D. Morris, H. Radha, CLOCs: camera-LiDAR object candidates fusion for 3D object detection, IEEE Int. Conf. Intell. Robot. Syst. (2020) 10386–10393, https://doi.org/10.1109/IROS45743.2020.9341791.
[64] F. Nobis, E. Shafiei, P. Karle, J. Betz, M. Lienkamp, Radar voxel fusion for 3D object detection, Appl. Sci. 11 (12) (Jun. 2021) 5598, https://doi.org/10.3390/ app11125598.
[65] R. Nabati, H. Qi, CenterFusion: center-based radar and camera fusion for 3d object detection, in: Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021, 2021, pp. 1526–1535, https://doi.org/10.1109/ WACV48630.2021.00157.
[66] K. Kundu, "nuScenes Detection Score," 2016. https://www.nuscenes.org/object-de tection?externalData=all&mapData=all&modalities=Any (accessed Dec. 06, 2021).
[67] N.-A.-M. Mai, P. Duthon, L. Khoudour, A. Crouzil, and S.A. Velastin, "Sparse LiDAR and stereo fusion (SLS-fusion) for depth estimation and 3D object detection," pp. 150–156, Mar. 2021, doi: 10.1049/icp.2021.1442.
[68] Y. You, et al., Pseudo-LiDAR++: accurate depth for 3D object detection in autonomous driving, ICLR (Jun. 2019). *2020*Accessed: Sep. 08, 2021. [Online]. Available, http://arxiv.org/abs/1906.06310.
[69] M. Zhu, C. Ma, P. Ji, X. Yang, Cross-modality 3d object detection, in: Proceedings 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021, 2021, pp. 3771–3780, https://doi.org/10.1109/WACV48630.2021.00382.
[70] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." pp. 770–778, 2016, Accessed: Dec. 06, 2021. [Online]. Available: http://image-n et.org/challenges/LSVRC/2015/.
[71] X.T. Vo, T.D. Tran, D.L. Nguyen, K.H. Jo, Stair-step feature pyramid networks for object detection, Commun. Comput. Inf. Sci. 1405 (2021) 168–175, https://doi. org/10.1007/978-3-030-81638-4\_13.
[72] M. Liang, B. Yang, S. Wang, R. Urtasun, Deep continuous fusion for multi-sensor 3D object detection, in: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11220, 2018, pp. 663–678, https://doi.org/10.1007/978-3-030-01270-0\_39. LNCS.
[73] S. Wang, S. Suo, W.C. Ma, A. Pokrovsky, R. Urtasun, Deep parametric continuous convolutional neural networks, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018, pp. 2589–2597, https://doi.org/10.1109/CVPR.2018.00274.
[74] Y. Wang, W.L. Chao, Di. Garg, B. Hariharan, M. Campbell, K.Q. Weinberger, Pseudo-lidar from visual depth estimation: bridging the gap in 3D object detection for autonomous driving, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019, pp. 8437–8445, https://doi. org/10.1109/CVPR.2019.00864, 2019-June.
[75] K.Q. Weinberger, B.D. Packer, L.K. Saul, Nonlinear dimensionality reduction by semidefinite programming and kernel matrix factorization, in: AISTATS 2005 - Proc. 10th Int. Work, 2005, pp. 381–388. *Artif. Intell. Stat.*Accessed: Dec. 03, 2021. [Online]. Available, http://proceedings.mlr.press/r5/weinberger05a/weinberge r05a.pdf.
[76] R. Qian, et al., End-to-End pseudo-LiDAR for image-based 3D object detection, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (Apr. 2020) 5880–5889, https://doi.org/10.1109/CVPR42600.2020.00592.
[77] S. Vora, A.H. Lang, B. Helou, O. Beijbom, Pointpainting: sequential fusion for 3D object detection, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 4603–4611, https://doi.org/ 10.1109/CVPR42600.2020.00466.

ghasemieh3dobjectdetectionautonomous2022.md 134 KB Түүх Анхны өгөгдөл