TY - JOUR
T1 - SOLSTM
T2 - Multisource Information Fusion Semantic Segmentation Network Based on SAR-OPT Matching Attention and Long Short-Term Memory Network
AU - Chang, Hao
AU - Fu, Xiongjun
AU - Guo, Kunyi
AU - Dong, Jian
AU - Guan, Jialin
AU - Liu, Chuyi
N1 - Publisher Copyright:
© 2004-2012 IEEE.
PY - 2025
Y1 - 2025
N2 - With the significant advancements in deep learning technology and the substantial improvement in remote sensing image resolution, remote sensing semantic segmentation has garnered widespread attention. Synthetic aperture radar (SAR) and optical images are the primary sources of remote sensing data, offering complementary information. SAR images can capture surface information even under cloud cover and at night, whereas optical images provide higher resolution in clear weather conditions. Deep learning-based feature fusion methods can effectively integrate multisource information to obtain more comprehensive surface data. However, there are significant spatiotemporal differences in multisource information, making it challenging to select and extract the most discriminative features for segmentation tasks. To address this, we propose a lightweight and efficient fusion semantic segmentation network, SOLSTM, which mixes SAR and optical images as inputs and performs cyclic cross-fusion to establish a new network paradigm. To tackle multisource data heterogeneity, we introduce SAR-OPT matching attention, which aggregates multisource image features by adaptively adjusting fusion weights, thereby achieving comprehensive perception of feature channels and contextual information. Additionally, to mitigate the high computational complexity of processing multidimensional data, we introduce the mLSTM block, which employs linear operations to mine global contextual information in fused images, thus reducing computational complexity and enhancing image segmentation performance. Experiments on the WHU-OPT-SAR dataset show that SOLSTM has excellent performance, achieving up to 52.9 mIoU and outperforming single source image segmentation, verifying the effective fusion of OPT-SAR.
AB - With the significant advancements in deep learning technology and the substantial improvement in remote sensing image resolution, remote sensing semantic segmentation has garnered widespread attention. Synthetic aperture radar (SAR) and optical images are the primary sources of remote sensing data, offering complementary information. SAR images can capture surface information even under cloud cover and at night, whereas optical images provide higher resolution in clear weather conditions. Deep learning-based feature fusion methods can effectively integrate multisource information to obtain more comprehensive surface data. However, there are significant spatiotemporal differences in multisource information, making it challenging to select and extract the most discriminative features for segmentation tasks. To address this, we propose a lightweight and efficient fusion semantic segmentation network, SOLSTM, which mixes SAR and optical images as inputs and performs cyclic cross-fusion to establish a new network paradigm. To tackle multisource data heterogeneity, we introduce SAR-OPT matching attention, which aggregates multisource image features by adaptively adjusting fusion weights, thereby achieving comprehensive perception of feature channels and contextual information. Additionally, to mitigate the high computational complexity of processing multidimensional data, we introduce the mLSTM block, which employs linear operations to mine global contextual information in fused images, thus reducing computational complexity and enhancing image segmentation performance. Experiments on the WHU-OPT-SAR dataset show that SOLSTM has excellent performance, achieving up to 52.9 mIoU and outperforming single source image segmentation, verifying the effective fusion of OPT-SAR.
KW - Lightweight network
KW - multisource fusion
KW - remote sensing
KW - semantic segmentation
KW - synthetic aperture radar (SAR)
KW - terrain classification
UR - http://www.scopus.com/inward/record.url?scp=85216731929&partnerID=8YFLogxK
U2 - 10.1109/LGRS.2025.3535524
DO - 10.1109/LGRS.2025.3535524
M3 - Article
AN - SCOPUS:85216731929
SN - 1545-598X
VL - 22
JO - IEEE Geoscience and Remote Sensing Letters
JF - IEEE Geoscience and Remote Sensing Letters
M1 - 4004705
ER -