Local Heterologous Virtual-to-Real Registration of The Body

This article focuses on the registration between digital local human body models and real local human bodies, which is a fundamental task in XR digital twins, intelligent cockpits, and other fields. Our proposed method enables adaptive registration of human bodies without the need for prior parametric models and external auxiliary sensors, only requiring a binocular camera. The method consists of two steps: coarse registration and fine registration. In the coarse registration step, it relies on a deep network to extract keypoints of the human bodies in RGB images and then performs anchor point registration with pre-set keypoints on the 3D digital model. In the fine registration step, it relies on deep learning-based segmentation of human bodies which can be recovered into real human body point clouds combining the pixel depth values computed by binocular camera. The segmentation of human bodies performs fine registration with the mesh vertex point clouds of the digital model scaled and free degrees fixed. In the pipeline, techniques such as lightweight design of deep networks, optimal initial values for fine registration through coarse registration and early stopping thresholds for iterations are used to reduce computational complexity and runtime while maintaining high accuracy.