Uncertainty-Driven Spectral Compressive Imaging with Spatial-Frequency Transformer

Recently, learning-based Hyperspectral image (HSI) reconstruction methods have demonstrated promising performance. However, existing learning-based methods still face two issues. 1) They rarely consider both the spatial sparsity and inter-spectral similarity priors of HSI. 2) They treat all image regions equally, ignoring that texture-rich and edge regions are more difficult to reconstruct than smooth regions. To address these issues, we propose an uncertainty-driven HSI reconstruction method termed Specformer. Specifically, we first introduce a frequency-wise self-attention (FWSA) module, and combine it with a spatial-wise local-window self-attention (LWSA) module in parallel to form a Spatial-Frequency (SF) block. LWSA can guide the network to focus on the regions with dense spectral information, and FWSA can capture the inter-spectral similarity. Parallel design helps the network to model cross-window connections, and expand its receptive fields while maintaining linear complexity. We use SF-block as the main building block in a multi-scale U-shape network to form our Specformer. In addition, we introduce an uncertainty-driven loss function, which can reinforce the network’s attention to the challenging regions with rich textures and edges. Experiments on simulated and real HSI datasets show that our Specformer outperforms state-of-the-art methods with lower computational and memory costs. The code is available at https://github.com/bianlab/Specformer.