UAV Swarm Cooperative Search based on Scalable Multiagent Deep Reinforcement Learning with Digital Twin-Enabled Sim-to-Real Transfer

Pan Cao, Lei Lei*, Gaoqing Shen, Shengsuo Cai, Xiaojiao Liu, Xiaochang Liu, Shiying Tian

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Cooperative target search (CTS) technology is highly desirable in various multi-UAV applications. However, searching for unknown targets in a dynamic threatening environment is a challenging problem, especially for UAVs with limited sensing range and communication capabilities. Besides, traditional searching methods lack scalability and efficient collaboration among the UAV swarm in dynamic environments. In this work, a digital twin (DT)-enabled distributed CTS approach was presented for UAV swarms and achieving sim-to-real transfer. Specifically, a new scalable multi-agent reinforcement learning (MARL) based algorithm called SAMARL is adopted to improve effectiveness and adaptability, combining a multi-head attention mechanism. In SAMARL, a scalable observation space with graph representation and an environmental cognition map is designed to thoroughly consider the target search rate, area coverage, and safety assurance. Then, a DT-driven training framework is proposed to facilitate the continuous evolution of MARL models and address the tradeoff between training speed and environment fidelity. Furthermore, we innovatively develop a distributed UAV swarm digital twin cooperative target search validation system, including real flight control, communication simulation tools, and a 3D physics engine. Extensive simulations validate its superiority compared to state-of-the-art strategies. More importantly, we also conduct real-world flight experiments on different scale mission areas and UAV swarms, further demonstrating the generalization and scalability of trained models.

Original languageEnglish
JournalIEEE Transactions on Mobile Computing
DOIs
Publication statusAccepted/In press - 2025
Externally publishedYes

Keywords

  • attention mechanism
  • Cooperative target search
  • digital twin
  • multi-agent proximal policy optimization
  • real-world experiments
  • UAV swarms

Fingerprint

Dive into the research topics of 'UAV Swarm Cooperative Search based on Scalable Multiagent Deep Reinforcement Learning with Digital Twin-Enabled Sim-to-Real Transfer'. Together they form a unique fingerprint.

Cite this

Cao, P., Lei, L., Shen, G., Cai, S., Liu, X., Liu, X., & Tian, S. (Accepted/In press). UAV Swarm Cooperative Search based on Scalable Multiagent Deep Reinforcement Learning with Digital Twin-Enabled Sim-to-Real Transfer. IEEE Transactions on Mobile Computing. https://doi.org/10.1109/TMC.2025.3530438