Enhanced applicability of reinforcement learning-based energy management by pivotal state-based Markov trajectories

Data- or sample-driven reinforcement learning (RL) is crucial for advancing AI models, enabling supervised learning-based AI to evolve autonomously. However, sample efficiency remains a key challenge, and simply increasing the number of training samples is not a guaranteed solution. More importantly, the focus should be on the breadth and diversity of the data distribution. This paper focuses on hybrid electric vehicles, with an emphasis on energy management. A novel training scheme for RL-based energy-saving policies is proposed, which relies on pivotal Markov transitions as state-based trajectories, significantly enhancing the adaptability of learning-based strategies. Firstly, the contradictions and limitations of the optimization terms in traditional reward functions are highlighted, including the misguidance of cumulative states and the cumbersome adjustment of weights. To address these issues, an unweighted reward is designed to simplify the training process and make it more universal. Secondly, the state-based featured driving cycle, as a novel concept, employs a 'question bank' style environment to expose the RL agent to a more diverse state space. Even with more sources and larger volumes of velocity data, the representative driving cycle can be condensed into customizable lengths of time domain, serving as the pivotal state-based Markov trajectory. Finally, after finishing offline training on the Tencent cloud server, an online driver-in-the-loop test is performed. The core advantage of the proposed strategy lies in completing the training in one go while offering greater applicability, aligning with the training concept more suitable for RL-based agents.