Skip to main navigation Skip to search Skip to main content

AirTF: Over-the-Air Token Fusion for Task-Oriented Multi-Modal Token Communications

  • Bole Liu
  • , Li Qiao
  • , Minghui Wu
  • , Yulin Shao
  • , Zhen Gao*
  • *Corresponding author for this work
  • Beijing Institute of Technology
  • The University of Hong Kong
  • Beijing Institute of Technology (Zhuhai)
  • CEMEE State Key Laboratory
  • MIIT Key Laboratory of Complex-Field Intelligent Sensing
  • Advanced Technology Research Institute (Jinan)
  • Yangtze Delta Region Academy of Bejing Institute of Technology
  • Xi'an University

Research output: Contribution to journal › Article › peer-review

Abstract

In the Internet of Vehicles (IoV), transmitting high-dimensional multi-modal sensory data to edge servers for time-sensitive tasks faces severe spectrum bottlenecks. To address this, we propose a foundation model-driven over-the-air token fusion (AirTF) framework for task-oriented multi-modal token communications. Unlike existing schemes for segmentation that rely on convolutional neural networks (CNNs) with limited local receptive fields, AirTF leverages vision transformer (ViT) encoders to extract globally contextualized semantic tokens from distributed heterogeneous sensors. By concurrently transmitting these spatially aligned tokens over a shared wireless channel, our framework exploits the superposition property of the multiple access channel to inherently fuse complementary multi-modal semantics (e.g., RGB and infrared) directly over the air. This mechanism significantly enhances spectral efficiency compared to orthogonal transmission. Furthermore, the integration of a pre-trained foundation model provides critical visual priors, effectively addressing the data-hungry nature of ViTs on limited, scenario-specific semantic segmentation datasets. Experiments demonstrate that AirTF consistently outperforms orthogonal transmission and CNN-based fusion baselines across AWGN and fading channels. Additional evaluations under a three-user setting, residual synchronization errors, and imperfect channel state information estimation further confirm its robustness.

Original languageEnglish
JournalIEEE Transactions on Vehicular Technology
DOIs
Publication statusAccepted/In press - 2026
Externally publishedYes

Keywords

  • Token communication
  • foundation models
  • multi-modality
  • task-oriented communication

Fingerprint

Dive into the research topics of 'AirTF: Over-the-Air Token Fusion for Task-Oriented Multi-Modal Token Communications'. Together they form a unique fingerprint.

Cite this