APL: Integrated Discriminative Features and Robust Boundary for Modulation Open-Set Recognition

As the electromagnetic environment becomes increasingly complex, traditional Automatic Modulation Recognition (AMR) methods cannot handle unknown modulation types that may arise under real-world conditions. Therefore, Automatic Modulation Open-Set Recognition (AMOSR) has gained significant attention as a technique that could identify unknown classes, playing a crucial role in enhancing the reliability of cognitive radio systems. Existing AMOSR approaches have primarily concentrated on either extracting discriminative features or establishing robust decision boundaries to enhance the AMOSR performance. To overcome existing limitation, we propose an Adversarial Prototype Learning (APL) algorithm to jointly optimize the features and boundaries through iterative refinement of prototype learning and adversarial learning. The prototype learning exploits the semantic similarity with the designed distribution distance-based cross entropy loss, aiming to obtain compact feature distributions while enhancing inter-class separability. The adversarial learning fully leverages the generated counterfactual images to constrain the unknown feature space, thus conducive to robust decision boundaries between known and unknown classes. The joint optimization yields mutual enhancements to boost AMOSR performance, as discriminative features facilitate boundary formulation, while well-defined boundaries further consolidate feature concentration. Comprehensive experiments on simulated and real-world signals demonstrate the effectiveness and robustness of APL compared to state-of-the-art AMOSR methods across various signal conditions.