Reinforcement Learning-based Real-time Energy Management for Plug-in Hybrid Electric Vehicle with Hybrid Energy Storage System

Energy allocation is a crucial issue for the energy storage system(ESS) of a plug-in hybrid electric vehicle (PHEV).In this paper, in order to realize an optimal energy allocation between the battery and the ultracapacitor in an ESS, a reinforcement learning-based real-time energy-management strategy was proposed. Firstly, a long driving condition which included various speed variations was chosen and the power transition probability matrices based Markov chain were calculated. Then, the reinforcement learning algorithm was used to obtain a control strategy aiming at minimizing the energy loss of the energy storage system. To use effectively the control strategy, the power transition probability matrices needed updating because the validation driving condition was different from the calculated driving condition and Kullback-Leibler(KL) divergence can be used to determine when the updating happened. At the same time, the updating-online control strategy was applied to the validation driving condition. Finally a comparison among the online energy management, offline energy management and the dynamic programming-based energy management was shown and the results indicate that the RL-based real-time energy-management strategy can decrease the energy loss and can be employed in real-time.