Virtual-to-Real Knowledge Transfer for Driving Behavior Recognition: Framework and a Case Study

Considering the difficulty and high cost of collecting sufficient data in the real world, driving simulators are used in many studies as an alternative data source, which can provide a much easier and safer way to collect driving data. However, because of the inherent differences between the virtual and real world, the recognition model for driving behavior trained using simulation-based data cannot fit the real driving scenes well. To fill the gap between simulation and real data, a knowledge transfer framework is proposed in this paper. Two transfer learning (TL) methods namely semi-supervised manifold alignment (SMA) and kernel manifold alignment (KEMA) are used in the proposed framework to map the data collected from the virtual and real world to one latent common space. A typical lane-changing scenario is selected for a case study. Three classifiers are trained in the latent space and used to do the lane-changing behavior recognition in the real world. In this way, sufficient simulation data are transferred to supplement the training set with few labeled real data, and thus improve the performance of behavior recognition in the real world. Compared with the traditional methods without knowledge transfer, classifiers combined with TL can reduce the error rate of recognition from around 30% (when only the real data are used) or higher than 50% (when only the simulation data are used) to as low as 11%.