Energy Management for a Hybrid Electric Vehicle Based on Blended Reinforcement Learning with Backward Focusing and Prioritized Sweeping

As a fundamental task of hybrid electric vehicles (HEVs), energy management strategies are critical for improving the performance. This paper proposes a new queue-Dyna reinforcement learning (RL) algorithm based energy management strategy (EMS), which can substantially reduce the online learning time while guarantees the control performance compared with widely used Q-learning. To solve the existing problems of direct and indirect RL based EMSs, a blended RL algorithm, Dyna, is introduced first. By reusing the actual experience to construct a model online, Dyna integrates direct and indirect RL, and thus has the advantages of both. Furthermore, two novel strategies of backward focusing and prioritized sweeping are incorporated in the Dyna framework, developing the queue-Dyna algorithm. To the best of our knowledge, it's the first attempt of adopting queue-Dyna in the EMS of a HEV. A comparative simulation of direct RL, indirect RL, Dyna and queue-Dyna is implemented, and the results demonstrate the proposed algorithm achieves a great improvement in fast learning and maintains satisfied fuel consumption. At last, a hardware-in-the-loop experiment verified the real-time performance of the proposed EMS.