An End-to-End HRL-based Framework with Macro-Micro Adaptive Layer for Mixed On-Ramp Merging

On-ramp merging problem focuses on vehicle safety and traffic efficiency. It can be considered as a hierarchical planning scenario with free flow zone, preparation zone and merging zone. Previous researches consider Reinforcement Learning (RL) as a potential solution due to its general learning ability. However, flat-RL tends to consider the on-ramp merging problem as a whole, neglecting its hierarchical property and causing limited improvement. Instead, with temporal abstraction, Option-based Hierarchical Reinforcement Learning (HRL) is capable to solve complicated problem by using task decomposition, giving a hint to adapt various zones in on-ramp merging problem. We hence propose an HRL-based Macro-Micro Adaptive framework (HRL-MMA). In this end-to-end framework, a Macro-Micro Adaptive Layer (MMAL) provides both macroscopic traffic information and microscopic vehicle information to the framework. The macroscopic information aims to help the master of the framework to choose options of different capacities, while the latter guarantees the safety of merging. Extensive experiments involve both the state-of-the-art baselines and several variants of the proposed framework. Compared with the IDM model, the proposed HRL-MMA framework has a 46.98% increase on the network average velocity and a 59.16% improvement on the emergency braking rate, largely ameliorating the safety of the merging problem.