3D mesh colorization from a single image via geometry prior modulation

We propose a novel framework for colorizing 3D meshes from a single RGB image, utilizing a triplane-based representation that integrates both geometric and image features. Unlike traditional texture mapping or view-dependent neural rendering approaches, our method directly predicts per-vertex colors without requiring camera pose information. To capture geometric context, we extract features from an uncolored mesh using a point-based encoder and project them onto three orthogonal planes, aligning them with the image space. Simultaneously, semantic features are extracted from the input image using a vision transformer. These image features are decoded into a triplane representation using a transformer-based decoder, where mesh features modulate the attention and feed-forward mechanisms, enriching the representation with geometric and appearance cues. Each mesh vertex then samples the refined triplane via bilinear interpolation to obtain a descriptive feature, which is decoded into a view-independent RGB color. The model is trained using a combination of 2D photometric loss computed from renderings of the predicted and ground-truth colored meshes, and a 3D vertex color loss. At inference, the method operates using a single RGB image and an uncolored mesh, without requiring camera pose, generating a colored mesh in under one second. Experiments on standard benchmarks demonstrate that our approach produces high-quality and consistent per-vertex colorization, outperforming existing single-view methods in both visual fidelity and generalization.