Fractional Fourier Image Transformer for Multimodal Remote Sensing Data Classification

Xudong Zhao, Mengmeng Zhang, Ran Tao*, Wei Li, Wenzhi Liao, Lianfang Tian, Wilfried Philips

*此作品的通讯作者

科研成果: 期刊稿件文章同行评审

59 引用 (Scopus)

摘要

With the recent development of the joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) data, deep learning methods have achieved promising performance owing to their locally sematic feature extracting ability. Nonetheless, the limited receptive field restricted the convolutional neural networks (CNNs) to represent global contextual and sequential attributes, while visual image transformers (VITs) lose local semantic information. Focusing on these issues, we propose a fractional Fourier image transformer (FrIT) as a backbone network to extract both global and local contexts effectively. In the proposed FrIT framework, HSI and LiDAR data are first fused at the pixel level, and both multisource feature and HSI feature extractors are utilized to capture local contexts. Then, a plug-and-play image transformer FrIT is explored for global contextual and sequential feature extraction. Unlike the attention-based representations in classic VIT, FrIT is capable of speeding up the transformer architectures massively and learning valuable contextual information effectively and efficiently. More significantly, to reduce redundancy and loss of information from shallow to deep layers, FrIT is devised to connect contextual features in multiple fractional domains. Five HSI and LiDAR scenes including one newly labeled benchmark are utilized for extensive experiments, showing improvement over both CNNs and VITs.

源语言英语
页(从-至)2314-2326
页数13
期刊IEEE Transactions on Neural Networks and Learning Systems
35
2
DOI
出版状态已出版 - 1 2月 2024

指纹

探究 'Fractional Fourier Image Transformer for Multimodal Remote Sensing Data Classification' 的科研主题。它们共同构成独一无二的指纹。

引用此

Zhao, X., Zhang, M., Tao, R., Li, W., Liao, W., Tian, L., & Philips, W. (2024). Fractional Fourier Image Transformer for Multimodal Remote Sensing Data Classification. IEEE Transactions on Neural Networks and Learning Systems, 35(2), 2314-2326. https://doi.org/10.1109/TNNLS.2022.3189994