Battery-Aware Cooperative Merging Strategy of Connected Electric Vehicles Based on Reinforcement Learning with Hindsight Experience Replay

Cooperative control of connected and automated electric vehicles (CAEVs) offers a great potential for a safe, high-efficient, sustainable transportation system. Among them, coordinated on-ramp merging based on an on-ramp merging vehicle and a mainline facilitating is a hot spot to lighten the impact of shockwave on highway junctions. Reinforcement learning (RL) is a promising solution to address this problem for its strong adaptiveness and self-learning ability. The existing methods are dedicatedly designed to circumvent the sparse reward problem of long-term merging safety but restrict the optimization space and neglect individual costs, especially irreversible loss, such as battery aging. In this article, a hindsight experience replay RL coordinated freeway on-ramp merging frame is proposed to face the sparse reward problem in ramp merging problem and, meanwhile, co-optimize traffic efficiency, energy-saving, and battery health performance. Compared with the existing optimization-based strategy, it fundamentally avoids abundant expert knowledge and elaborate design for reward shaping and model construction while ensuring the nature of merging tasks without excessive simplification. Numerical experiments are conducted to verify the optimality, adaptability, and self-learning ability of the proposed strategy. With comparison experiment, the proposed strategy surpasses state-of-the-art RL methods more than 13% in overall index throughput with 12% energy consumption while easing individual battery aging during the merging process.