TY - GEN
T1 - Gaussian Replacement
T2 - 19th Chinese Conference on Image and Graphics Technologies and Applications, IGTA 2024
AU - Dongye, Xiaonuo
AU - Guo, Hanzhi
AU - Bao, Yihua
AU - Weng, Dongdong
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.
PY - 2025
Y1 - 2025
N2 - This paper presents the Gaussian Replacement, a novel system that integrates mesh representation and Gaussian representation for object rendering in virtual reality (VR). The system comprises two primary phases, i.e. the 3D Gaussian model preparation phase and the real-time interaction phase. During the 3D Gaussian model preparation phase, camera movements within a mesh environment are leveraged to generate multi-view images. These images are subsequently employed to train 3D Gaussian Splatting, facilitating the virtual scene represented with Gaussians. Object masks are obtained by utilizing instance segmentation and inpainting on the multi-view images to enable the extraction of Gaussian Replacement for objects within the mesh-represented scene. In the real-time interaction phase, a joint probability model is implemented, utilizing user factors and object factors to determine the likelihood of representing each object with either mesh or Gaussians. System evaluations validate the Gaussian Replacement system’s capacity to achieve realism and interactivity while simultaneously minimizing computational costs. The proposed system presents a promising avenue for enriching VR experiences through the joint rendering of mesh and 3D Gaussians in real-time VR interaction.
AB - This paper presents the Gaussian Replacement, a novel system that integrates mesh representation and Gaussian representation for object rendering in virtual reality (VR). The system comprises two primary phases, i.e. the 3D Gaussian model preparation phase and the real-time interaction phase. During the 3D Gaussian model preparation phase, camera movements within a mesh environment are leveraged to generate multi-view images. These images are subsequently employed to train 3D Gaussian Splatting, facilitating the virtual scene represented with Gaussians. Object masks are obtained by utilizing instance segmentation and inpainting on the multi-view images to enable the extraction of Gaussian Replacement for objects within the mesh-represented scene. In the real-time interaction phase, a joint probability model is implemented, utilizing user factors and object factors to determine the likelihood of representing each object with either mesh or Gaussians. System evaluations validate the Gaussian Replacement system’s capacity to achieve realism and interactivity while simultaneously minimizing computational costs. The proposed system presents a promising avenue for enriching VR experiences through the joint rendering of mesh and 3D Gaussians in real-time VR interaction.
KW - 3D Gaussian Splatting
KW - Joint Probability Model
KW - Real-time VR Interaction
UR - https://www.scopus.com/pages/publications/85214269992
U2 - 10.1007/978-981-97-9919-0_25
DO - 10.1007/978-981-97-9919-0_25
M3 - Conference contribution
AN - SCOPUS:85214269992
SN - 9789819799183
T3 - Communications in Computer and Information Science
SP - 312
EP - 326
BT - Image and Graphics Technologies and Applications - 19th Chinese Conference, IGTA 2024, Revised Selected Papers
A2 - Wang, Yongtian
A2 - Huang, Hua
PB - Springer Science and Business Media Deutschland GmbH
Y2 - 16 August 2024 through 18 August 2024
ER -