Attention-Guided Multi-modal and Multi-scale Fusion for Multispectral Pedestrian Detection

Multispectral pedestrian detection provides more accurate and reliable detection results by leveraging complementary information from color-thermal modalities and has drawn much attention in the open world. Much progress has been made in the feature-level-based detection methods which aim to effectively fuse the multispectral features extracted by the convolution neural networks. However, existing methods mainly focus on the information integration between the same-level feature maps and ignore the complementary local features scattered in multi-scale layers. In this paper, we introduce an Attention-guided multi-Modal and multi-Scale Fusion (AMSF) module to simultaneously sample complementary local features scattered in multi-modal and multi-scale layers, and adaptively aggregate them with fine-grained attention to fully exploit different modalities for better multi-scale detection results. Extensive experiments are conducted on three multispectral datasets and three representative deep-learning-based detection benchmarks to show the effectiveness and generalization of the proposed method, and the state-of-the-art detection performance.