Structure-aware hierarchical deep reinforcement learning for LEO satellite resource allocation

LEO satellite resource allocation in global satellite communication systems is inherently challenged by the complexity, diversity, and dynamic nature of user demands. To mitigate the gap in multi-satellite resource allocation for dynamic optimization of long-term performance, we propose a structure-aware hierarchical deep reinforcement learning framework, which integrates dual-layer optimization with temporal dynamics modeling and the system structure parameters extraction. In the upper level, the global satellite coordination layer employs a collaborative DQN to optimize inter-satellite handover processes, ensuring system-wide resource allocation equilibrium at each time step. In the lower level, the local resource optimization layer focuses on intra-satellite resource management by structure-aware DQN method. The proposed network autonomously perceives system structure parameters from input data matrix for sequence state space modeling. The structure-aware result is incorporated into the temporal feature extraction process of DQN for the fine-grained optimization of beam resources. Compared with traditional DQN method, hierarchical DQN (HDQN) method, and the transformer-based hierarchical DQN (THDQN) method, the proposed structure-aware hierarchical DQN algorithm (denoted as SHDQN) in experiments demonstrates improvements in both system-level resource utilization efficiency and network training time. The average number of users whose demands are satisfied by SHDQN increases by 10.03% and 20.67% compared to THDQN and HDQN with fewer epochs, respectively, improving resource allocation performance and saving on-board computing time for large-scale dynamic satellite systems.