UAV Swarm Cooperative Search based on Scalable Multiagent Deep Reinforcement Learning with Digital Twin-Enabled Sim-to-Real Transfer

Cooperative target search (CTS) technology is highly desirable in various multi-UAV applications. However, searching for unknown targets in a dynamic threatening environment is a challenging problem, especially for UAVs with limited sensing range and communication capabilities. Besides, traditional searching methods lack scalability and efficient collaboration among the UAV swarm in dynamic environments. In this work, a digital twin (DT)-enabled distributed CTS approach was presented for UAV swarms and achieving sim-to-real transfer. Specifically, a new scalable multi-agent reinforcement learning (MARL) based algorithm called SAMARL is adopted to improve effectiveness and adaptability, combining a multi-head attention mechanism. In SAMARL, a scalable observation space with graph representation and an environmental cognition map is designed to thoroughly consider the target search rate, area coverage, and safety assurance. Then, a DT-driven training framework is proposed to facilitate the continuous evolution of MARL models and address the tradeoff between training speed and environment fidelity. Furthermore, we innovatively develop a distributed UAV swarm digital twin cooperative target search validation system, including real flight control, communication simulation tools, and a 3D physics engine. Extensive simulations validate its superiority compared to state-of-the-art strategies. More importantly, we also conduct real-world flight experiments on different scale mission areas and UAV swarms, further demonstrating the generalization and scalability of trained models.