3D Object Detection and Tracking Based on Lidar-Camera Fusion and IMM-UKF Algorithm Towards Highway Driving

In this work, we propose DTFI: a 3D object Detection and Tracking approach consisting of lidar-camera Fusion-based 3D object detection and Interacting multiple model with unscented Kalman filter (IMM-UKF) based tracking algorithm towards highway driving. For the 3D object detection, an end-to-end learnable architecture fuses the point cloud from the lidar and images from the camera to generate 3D bounding boxes. First, the point cloud is augmented by the full-resolution image-only feature maps in the architecture. In addition, to address the misalignment of raw data due to calibration errors, windowed convolution is used to match multimodal data. Then, new features are extracted from the augmented point cloud for generating bounding boxes. Moreover, to achieve real-time performance in highway scenarios, a fast encoding network is used to extract features. With the object detection results, multiple object tracking (MOT) is performed, which includes object state estimation and data association. The IMM-UKF algorithm is proposed to estimate the 3D space state to deal with the movement uncertainty of the objects, and the Hungarian algorithm is used for data association. Furthermore, to set the appropriate covariance and the state switching matrix for specific scenarios, the Particle Swarm Optimization (PSO) algorithm is used. To evaluate the performance of the proposed DTFI, comparisons with related methods are made on the KITTI testing benchmark. Experimental results show that the proposed DTFI outperforms the existing fusion-based detection and tracking methods at 30 Hz, which can meet the essential requirement of highway scenarios.