Enhanced Spectral–Spatial Fusion Network for Multispectral Object Detection in Ground-Aerial Images

In recent years, multispectral object detection technology has gained widespread attention due to its significant performance in detecting objects with similar colors or textures in complex environments. The mainstream methods adopt the fusion of visible light (RGB) and thermal images to make up for the shortcomings of a single modality, improving detection accuracy. However, most methods fail to take the inherent differences between modalities into account. RGB and thermal images differ in object attributes, which may lead to inconsistent contribution of each modality to the fusion features. Hence, equally feeding them into the feature extractor will limit the expressiveness of the fusion features. In order to reasonably utilize complementary information cues of each modality, an effective cross-modality feature fusion network is proposed in this letter. It comprises a spectral–spatial enhancing (SSE) module and a feature fusion module via Transformer [fast Fourier transform (FFT)]. For dual-modality data, in the aerial dataset, our model’s detection accuracy metrics, mAP50 and mean average precision (mAP), are, respectively, improved by 1.2% and 0.9% compared to the best dual-modality network. Comprehensive experiments on both ground and aerial datasets demonstrate that our approach outperforms existing methods. The achievements are of great significance for enhancing the robustness and accuracy of multispectral object detection.