Gaussian Replacement: Gaussians-Mesh Joint Rendering for Real-Time VR Interaction

This paper presents the Gaussian Replacement, a novel system that integrates mesh representation and Gaussian representation for object rendering in virtual reality (VR). The system comprises two primary phases, i.e. the 3D Gaussian model preparation phase and the real-time interaction phase. During the 3D Gaussian model preparation phase, camera movements within a mesh environment are leveraged to generate multi-view images. These images are subsequently employed to train 3D Gaussian Splatting, facilitating the virtual scene represented with Gaussians. Object masks are obtained by utilizing instance segmentation and inpainting on the multi-view images to enable the extraction of Gaussian Replacement for objects within the mesh-represented scene. In the real-time interaction phase, a joint probability model is implemented, utilizing user factors and object factors to determine the likelihood of representing each object with either mesh or Gaussians. System evaluations validate the Gaussian Replacement system’s capacity to achieve realism and interactivity while simultaneously minimizing computational costs. The proposed system presents a promising avenue for enriching VR experiences through the joint rendering of mesh and 3D Gaussians in real-time VR interaction.