Addressing Hybrid Action of BVR Engagement With Joint Policy Optimization

Multi-agent reinforcement learning (MARL) has attracted increasing attention in the context of beyond-visual-range (BVR) engagements involving unmanned combat aerial vehicles (UCAVs) in recent years. UCAVs are required to make firing and maneuvering decisions simultaneously, while maneuvers also encompass both target-independent and target-dependent categories, resulting in a hybrid action space. This paper proposes joint policy optimization (JPO) to address the challenges associated with learning within such a hybrid action space. Heterogeneous Transformer is developed to fuse information from diverse battlefield observations, thereby overcoming the limitation inherent in the uniform weight matrices of conventional Transformers. Furthermore, a supervision enhancement technique is introduced, enabling agents to learn missile hit probability during self-play. This method allows UCAVs to launch missiles when the estimated hit probability surpasses a predefined threshold. For maneuvering decisions, a hybrid soft actor-critic is proposed to jointly optimize the action spaces corresponding to different maneuver types, enabling UCAVs to select both target-independent and target-dependent maneuvers from the joint policy. A theoretical entropy setting over the hybrid maneuver space is also given. Simulation results demonstrate that the proposed method outperforms existing MARL algorithms.