Geometry-guided multilevel RGBD fusion for surface normal estimation

Developments in 3D computer vision have advanced scene understanding and 3D modeling. Surface normal estimation is a basic task in these fields. In this paper, we propose a geometry-guided multilevel fusion scheme for high-quality surface normal estimation by exploiting texture and geometry information from color and depth images. The surface normal is progressively predicted with a coarse-to-fine strategy. First, an initial surface normal (IniNormal) N_ini is predicted by a hierarchical confidence reweighting convolution neural network to merge texture and geometry information in a CNN feature level. Although a general accuracy is achieved, the long tail problem makes the IniNormal always fails in special areas where the depth map is high-quality while the intensity interference is challenging, such as repeating textures and abnormal exposures. Further, a traditional geometry-consistent based surface normal(GeoNormal) N_geo is calculated based on traditional constraints, and a surface normal level fusion module is designed to remap the depth to different representations and reconsider scene information. Then, the final clear surface normal N is estimated by adaptively reintegrating the IniNormal and GeoNormal in a decision level. To overcome disturbances in the dataset and ensure the trainability of the network, a carefully designed hybrid objective function and an annealed term are applied. An explainable analysis is attached. The experimental results on two benchmark datasets demonstrate that the proposed GMLF(geometry-guided multilevel RGBD fusion for surface normal estimation) can achieve better quantitative and qualitative performance. The proposed method may be useful for robots and auto-driving which can be applied in the next-generation Internet-of-Things (NG-IoT).