Pretrained Self-supervised Material Reflectance Estimation Based on a Differentiable Image-Based Renderer

Measuring the material reflectance of surfaces is a key technology in inverse rendering, which can be used in object appearance reconstruction. In this paper we propose a novel deep learning-based method to extract material information represented by a physically-based bidirectional reflectance distribution function from an RGB image of an object. Firstly, we design new deep convolutional neural network architectures to regress material parameters by self-supervised training based on a differentiable image-based renderer. Then we generate a synthetic dataset to train the model as the initialization of the self-supervised system. To transfer the domain from the synthetic data to the real image, we introduce a test-time training strategy to finetune the pretrained model to improve the performance. The proposed architecture only requires one image as input and the experiments are conducted to evaluate the proposed method on both the synthetic data and real data. The results show that our trained model presents dramatic improvement and verifies the effectiveness of the proposed methods.