Online-Learning Adaptive Energy Management for Hybrid Electric Vehicles in Various Driving Scenarios Based on Dyna Framework

The practical driving scenarios have a decisive influence on the performance of energy management strategies (EMSs) for hybrid electric vehicles (HEVs). How to adapt to various scenarios is still a challenging task for reinforcement learning (RL)-based EMSs. To tackle this problem, this article proposes an online-learning adaptive EMS based on an RL framework, Dyna. In the framework, besides directly improving the policy, the real experience from the vehicle is reused to establish an interactive model. Through the model, simulated experience is generated to meanwhile improve the policy and most high-priced real-world exploration is substituted by model-based exploration, which substantially reduces the learning time and lowers the learning price. Thus, Dyna-based EMS achieves fast and low-cost online learning when the HEV enters a new scenario, significantly enhancing the strategy's adaptability. In the simulation, three typical driving scenarios are first selected for testing: city, suburb, and highway. For all three scenarios, Dyna only cumulates about 60% driving cost of deep Q-network (DQN) in the first 100 cycles. In an off-road scenario, Dyna also achieves about 70% cumulative costs compared with deterministic policy gradient (DDPG) and twin delayed deep deterministic policy gradient (TD3) considering 100 cycles, which demonstrates the superiority in the adaptability of the proposed EMS.