PointGPTv2: Universal Auto-regressively Generative Pre-training Framework for Point Clouds

Large language models (LLMs) based on the generative pre-training transformer (GPT) have demonstrated remarkable effectiveness across a diverse range of downstream tasks. Inspired by the progress of GPT, we present PointGPTv2, a universal framework that extends the concept of GPT to point clouds. To this end, a point cloud auto-regressive generation task is proposed to pre-train transformer models. Our method divides the input point cloud into point patches and orders them into a sequence according to their spatial proximity. Then, a hierarchical extractor, with the causal fusion module, learns multi-scale latent representations conditioned on the preceding point patches, aiming to predict the subsequent point patch in an auto-regressive manner. During fine-tuning, the geo-adapter is inserted so that the model better captures fine-grained geometric features. To fully unleash the power of PointGPTv2, we collect larger pre-training datasets and add a subsequent postpre- training stage. Experiments demonstrate that our scalable approach allows for learning high-capacity models that generalize well, achieving state-of-the-art (SOTA) performance on various downstream tasks. In particular, our approach achieves classification accuracies of 94.9% on the ModelNet40 dataset and 93.6% on the ScanObjectNN dataset, outperforming all other single-modal pre-training methods. Moreover, we also exhibit outstanding performance on LiDAR point cloud tasks, achieving 76.96 mAP (L2) on Waymo and 66.58 mAP on ONCE.