Realistic Fusion of Virtual Materials and Hands for AR through Intrinsic Decomposition

Intrinsic decomposition is an inherent issue in computer vision and graphics, which usually decomposes the image into reflectance and shading. Reflectance and shading can be applied to achieve virtual and real information fusion with illumination consistency in augmented reality, but this demands higher performance and speed of the intrinsic decomposition method. In this study, we propose a novel neural network architecture for intrinsic decomposition that enables real-time generation of high-quality reflectance and shading. Furthermore, we extend our framework by integrating the hands segmentation network, resulting in an augmented reality framework capable of seamlessly fusing virtual materials and hands. Through comprehensive experiments, we demonstrate the effectiveness of our framework in achieving realistic fusion of multiple virtual materials and hands, even in the presence of dynamic gestures and diverse illumination conditions. We believe that our study will serve as an inspiration for the development of more creative applications in fields such as art, film, and advertising.