TY - JOUR
T1 - UAV Swarm Cooperative Search based on Scalable Multiagent Deep Reinforcement Learning with Digital Twin-Enabled Sim-to-Real Transfer
AU - Cao, Pan
AU - Lei, Lei
AU - Shen, Gaoqing
AU - Cai, Shengsuo
AU - Liu, Xiaojiao
AU - Liu, Xiaochang
AU - Tian, Shiying
N1 - Publisher Copyright:
© 2002-2012 IEEE.
PY - 2025
Y1 - 2025
N2 - Cooperative target search (CTS) technology is highly desirable in various multi-UAV applications. However, searching for unknown targets in a dynamic threatening environment is a challenging problem, especially for UAVs with limited sensing range and communication capabilities. Besides, traditional searching methods lack scalability and efficient collaboration among the UAV swarm in dynamic environments. In this work, a digital twin (DT)-enabled distributed CTS approach was presented for UAV swarms and achieving sim-to-real transfer. Specifically, a new scalable multi-agent reinforcement learning (MARL) based algorithm called SAMARL is adopted to improve effectiveness and adaptability, combining a multi-head attention mechanism. In SAMARL, a scalable observation space with graph representation and an environmental cognition map is designed to thoroughly consider the target search rate, area coverage, and safety assurance. Then, a DT-driven training framework is proposed to facilitate the continuous evolution of MARL models and address the tradeoff between training speed and environment fidelity. Furthermore, we innovatively develop a distributed UAV swarm digital twin cooperative target search validation system, including real flight control, communication simulation tools, and a 3D physics engine. Extensive simulations validate its superiority compared to state-of-the-art strategies. More importantly, we also conduct real-world flight experiments on different scale mission areas and UAV swarms, further demonstrating the generalization and scalability of trained models.
AB - Cooperative target search (CTS) technology is highly desirable in various multi-UAV applications. However, searching for unknown targets in a dynamic threatening environment is a challenging problem, especially for UAVs with limited sensing range and communication capabilities. Besides, traditional searching methods lack scalability and efficient collaboration among the UAV swarm in dynamic environments. In this work, a digital twin (DT)-enabled distributed CTS approach was presented for UAV swarms and achieving sim-to-real transfer. Specifically, a new scalable multi-agent reinforcement learning (MARL) based algorithm called SAMARL is adopted to improve effectiveness and adaptability, combining a multi-head attention mechanism. In SAMARL, a scalable observation space with graph representation and an environmental cognition map is designed to thoroughly consider the target search rate, area coverage, and safety assurance. Then, a DT-driven training framework is proposed to facilitate the continuous evolution of MARL models and address the tradeoff between training speed and environment fidelity. Furthermore, we innovatively develop a distributed UAV swarm digital twin cooperative target search validation system, including real flight control, communication simulation tools, and a 3D physics engine. Extensive simulations validate its superiority compared to state-of-the-art strategies. More importantly, we also conduct real-world flight experiments on different scale mission areas and UAV swarms, further demonstrating the generalization and scalability of trained models.
KW - attention mechanism
KW - Cooperative target search
KW - digital twin
KW - multi-agent proximal policy optimization
KW - real-world experiments
KW - UAV swarms
UR - http://www.scopus.com/inward/record.url?scp=85215436846&partnerID=8YFLogxK
U2 - 10.1109/TMC.2025.3530438
DO - 10.1109/TMC.2025.3530438
M3 - Article
AN - SCOPUS:85215436846
SN - 1536-1233
JO - IEEE Transactions on Mobile Computing
JF - IEEE Transactions on Mobile Computing
ER -