A Progressive Approach to Learn Global and Local Multi-view Features for 3D Visual Grounding

The 3D visual grounding task aims to localize objects in point clouds based on natural language descriptions, playing a significant role in various domains such as autonomous driving and augmented reality. In this task, the view inconsistency between textual observation perspectives and point cloud viewpoints causes view confusion problems that hinder the model’s ability to accurately localize target objects. To address this issue, this paper proposes a progressive multi-view feature approach to supplement point cloud information from different perspectives, which includes sequential-form global multi-view features and vector-form local multi-view features. This method progressively learns multi-view point cloud features within the model while designing explicit interaction between object relative positions and textual descriptions to enhance the model’s comprehension of multimodal information. Furthermore, we introduce a selective state space model as the learning module for sequential global multi-view features, which improves model accuracy while reducing memory consumption and training time. Experimental results demonstrate that the proposed method achieves superior performance over existing state-of-the-art approaches on public datasets.