A hierarchical multiscale reinforcement learning framework for energy management in hybrid electric vehicles considering powertrain response coordination

Reinforcement learning (RL) demonstrates significant potential for enhancing the performance of energy management strategies (EMSs) in hybrid electric vehicles (HEVs). However, most existing studies focus primarily on macroscopic power allocation and give insufficient consideration to the response capability of the powertrain, especially the coupled electromechanical dynamics of the engine-generator set (EGS). To address this limitation, this paper proposes a hierarchical multiscale RL framework. The upper layer employs a twin delayed deep deterministic policy gradient (TD3)-based macroscopic EMS at a slow timescale to determine the EGS reference operating point, which defines the reference power. By incorporating engine speed fluctuations into the optimization objective, it achieves a synergistic trade-off between fuel economy and transient smoothness. The lower layer employs a multiscale dual-agent RL strategy to resolve the dynamic bandwidth mismatch within the EGS. Through a temporally asynchronous update mechanism, it explicitly synchronizes engine torque build-up with generator dynamic loading. To mitigate cross-scale distribution shifts and joint-training convergence difficulties, a cooperative training environment is constructed, integrating an explicit-duration hidden semi-markov model with optimal transport (EHSMM-OT) and a task-driven curriculum control strategy (TCCS). Hardware-in-the-loop (HIL) validation demonstrates that this framework accelerates convergence by 15%. Under stochastic drive cycles, fuel consumption deviation from the dynamic programming (DP) benchmark is strictly constrained within 3.50%. Crucially, while achieving coordinated control performance comparable to model predictive control (MPC), the proposed framework reduces the cumulative computation time by 57.50%, demonstrating substantially improved computational efficiency and favorable real-time implementation capability. Overall, the proposed framework achieves favorable fuel economy, powertrain response performance, and engineering deployability.