Military Image Captioning for Low-Altitude UAV or UGV Perspectives

Lizhi Pan, Chengtian Song*, Xiaozheng Gan, Keyu Xu, Yue Xie

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Low-altitude unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs), which boast high-resolution imaging and agile maneuvering capabilities, are widely utilized in military scenarios and generate a vast amount of image data that can be leveraged for textual intelligence generation to support military decision making. Military image captioning (MilitIC), as a visual-language learning task, provides innovative solutions for military image understanding and intelligence generation. However, the scarcity of military image datasets hinders the advancement of MilitIC methods, especially those based on deep learning. To overcome this limitation, we introduce an open-access benchmark dataset, which was termed the Military Objects in Real Combat (MOCO) dataset. It features real combat images captured from the perspective of low-altitude UAVs or UGVs, along with a comprehensive set of captions. Furthermore, we propose a novel encoder–augmentation–decoder image-captioning architecture with a map augmentation embedding (MAE) mechanism, MAE-MilitIC, which leverages both image and text modalities as a guiding prefix for caption generation and bridges the semantic gap between visual and textual data. The MAE mechanism maps both image and text embeddings onto a semantic subspace constructed by relevant military prompts, and augments the military semantics of the image embeddings with attribute-explicit text embeddings. Finally, we demonstrate through extensive experiments that MAE-MilitIC surpasses existing models in performance on two challenging datasets, which provides strong support for intelligence warfare based on military UAVs and UGVs.

Original languageEnglish
Article number421
JournalDrones
Volume8
Issue number9
DOIs
Publication statusPublished - Sept 2024

Keywords

  • image understanding
  • military image captioning
  • unmanned aerial vehicle (UAV)
  • unmanned ground vehicle (UGV)
  • visual-language model

Fingerprint

Dive into the research topics of 'Military Image Captioning for Low-Altitude UAV or UGV Perspectives'. Together they form a unique fingerprint.

Cite this