Functional-Penetrated Interactive System Towards Virtual-Real Fusion Environments

Current mixed reality technology cannot completely map the information in the physical world to the virtual environment, and the user must take off the head-mounted display to deal with the affairs in the real environment when using the mixed reality system. This greatly reduces the immersion created by the virtual reality system, and also reduces work efficiency. In order to realize the user's direct interaction with the physical world in the virtual environment without destroying the immersion of the virtual reality system, this paper proposes a functional-penetrated interactive system towards the virtual-real fusion environment, which extracts the visual information of objects from the real scene captured by the RGB camera through deep neural networks, completes virtual-real registration and presents it in real time in the virtual environment, and finally enables the user to complete the manipulation of the physical objects in the virtual environment. In addition, the system also allows the user to dynamically adjust the virtual-real fusion ratio according to the user's needs, which achieves a balance between the immersion of the virtual-real fusion environment and the interaction efficiency of the system. Experimental results show that the system can provide users with more realistic physical feedback, with higher interaction freedom and universality.