Unsupervised Domain Adaptation with Extended Multimodal LSTM for End-to-End Classification of Multisource Remote Sensing Data

Deep learning has gained success in unsupervised domain adaptation (UDA) for remote sensing (RS) data classification. However, existing methods lack specialized deep learning structure components for multisource RS (MSRS) data, restricting their performance. As such, a cross-modal compensation-consistent U-shape masked extended multimodal long short-term memory neural network (CCU-MaxNet) framework is proposed for UDA in end-to-end classification of MSRS data. First, the extended multimodal long short-term memory (xMLSTM) cell is designed as a new module, with which its down-block and up-block are built to explore the complementarity of MSRS data and long-term dependencies between their hierarchical features, serving as memory state. Then, a cross-modal differential compensation (CMDC) block is devised to enhance the complementary modality-specific information. Based on these blocks, a U-MaxNet is built to extract the representational features under the guidance of the memory state. A multimodal compensation consistent domain adaptation network (MCC-DANet) is further designed for domain alignment at both pixel and semantic levels. Finally, by integrating U-MaxNet and MCC-DANet, CCU-MaxNet extracts more discriminative domain-invariant features for cross-domain classification. Particularly, the proposed framework has excellent flexibility, providing a high-precision mode and a high-efficiency mode based on whether to use the fractional Fourier transform (FrFT) to meet the requirements of different applications. Extensive experiments on four cross-domain MSRS datasets verify its superiority compared to state-of-the-art methods. Our source code will be available at https://github.com/WenshuaiHu/UDA-MSRS-CCU-MaxNet.