SpectralX: Parameter-efficient domain generalization for spectral Remote Sensing Foundation Models

Recent advances in Remote Sensing Foundation Models (RSFMs) have led to significant breakthroughs in the field. While many RSFMs have been pretrained with massive optical imagery, more multispectral/hyperspectral data remain lack of the corresponding foundation models. To leverage the advantages of spectral imagery in earth observation, we explore whether existing RSFMs can be effectively adapted to process diverse spectral modalities without requiring extensive spectral pretraining. In response to this challenge, we proposed SpectralX, an innovative parameter-efficient fine-tuning framework that adapt existing RSFMs as backbone while introducing a two-stage training approach to handle various spectral inputs, thereby significantly improving domain generalization performance. In the first stage, we employ a masked-reconstruction task and design a specialized Hyper Tokenizer (HyperT) to extract attribute tokens from both spatial and spectral dimensions. Simultaneously, we develop an Attribute-oriented Mixture of Adapter (AoMoA) that dynamically modulate features while performing layer-wise fine-tuning. With semantic segmentation as downstream task in the second stage, we plug an Attribute-refined Adapter (Are-adapter) behind AoMoA. By iteratively querying low-level semantic features with high-level representations, the model learns to focus on task-beneficial features, enabling customized adjustment of RSFMs. Following this two-phase adaptation process, SpectralX is capable of interpreting spectral imagery from new regions or seasons. We have collected three benchmark spectral datasets and constructed eight domain generalization tasks. Comprehensive experiments demonstrate that SpectralX effectively adapts to diverse spectral imagery and outperforms state-of-the-art methods in cross-domain interpretation tasks. SpectralX achieves average improvements of 4.2% mIoU and 3.5% mIoU over RSFMs under without and with domain-gap settings, respectively. The codes will be available from the website: https://github.com/YuxiangZhang-BIT/SpectralX.