Cross-Modal LiDAR-Visual-Inertial Localization in Prebuilt LiDAR Point Cloud Map Through Direct Projection

This article proposes the light detection and ranging (LiDAR) map LiDAR-visual-inertial localization (LM-LVIL) system, integrating LiDAR, visual, and inertial sensors within a preconstructed LiDAR map. The key innovation is cross-modal fusion between camera images and point cloud maps via direct projection. Unlike traditional methods aligning 3-D points triangulated from the camera with the point cloud, our system projects map points into the camera frame to minimize photometric error, similar to direct methods in visual simultaneous localization and mapping (SLAM). The system uses an iterated error state Kalman filter (IESKF) framework, incorporating inertial measurement unit (IMU) integration, visual measurements, and LiDAR measurement for precise pose estimation. LiDAR measurement aligns points by ensuring scan points lie on a local plane in the map. Visual measurements include photometric measurement, projection measurement, 3-D point alignment, and 2-D–3-D line measurement. The experimental results show that LiDAR map visual-inertial localization (LM-VIL) achieves competitive accuracy indoors compared to the state-of-the-art methods. The LM-LVIL system improves accuracy in both indoor and outdoor scenarios compared to the LiDAR map LiDAR-inertial localization (LM-LIL) system and the LiDAR localization baseline method. Our localization system balances accuracy and real-time performance, demonstrating application potential. (Figure presented).