An End-to-End HRL-based Framework with Macro-Micro Adaptive Layer for Mixed On-Ramp Merging

Zoutao Wen, Huachun Tan, Bo Yu, Yanan Zhao*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

On-ramp merging problem focuses on vehicle safety and traffic efficiency. It can be considered as a hierarchical planning scenario with free flow zone, preparation zone and merging zone. Previous researches consider Reinforcement Learning (RL) as a potential solution due to its general learning ability. However, flat-RL tends to consider the on-ramp merging problem as a whole, neglecting its hierarchical property and causing limited improvement. Instead, with temporal abstraction, Option-based Hierarchical Reinforcement Learning (HRL) is capable to solve complicated problem by using task decomposition, giving a hint to adapt various zones in on-ramp merging problem. We hence propose an HRL-based Macro-Micro Adaptive framework (HRL-MMA). In this end-to-end framework, a Macro-Micro Adaptive Layer (MMAL) provides both macroscopic traffic information and microscopic vehicle information to the framework. The macroscopic information aims to help the master of the framework to choose options of different capacities, while the latter guarantees the safety of merging. Extensive experiments involve both the state-of-the-art baselines and several variants of the proposed framework. Compared with the IDM model, the proposed HRL-MMA framework has a 46.98% increase on the network average velocity and a 59.16% improvement on the emergency braking rate, largely ameliorating the safety of the merging problem.

Original languageEnglish
Title of host publication35th IEEE Intelligent Vehicles Symposium, IV 2024
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1424-1429
Number of pages6
ISBN (Electronic)9798350348811
DOIs
Publication statusPublished - 2024
Event35th IEEE Intelligent Vehicles Symposium, IV 2024 - Jeju Island, Korea, Republic of
Duration: 2 Jun 20245 Jun 2024

Publication series

NameIEEE Intelligent Vehicles Symposium, Proceedings
ISSN (Print)1931-0587
ISSN (Electronic)2642-7214

Conference

Conference35th IEEE Intelligent Vehicles Symposium, IV 2024
Country/TerritoryKorea, Republic of
CityJeju Island
Period2/06/245/06/24

Keywords

  • Connected and Automated Vehicles (CAVs)
  • hierarchical reinforcement learning (HRL)
  • on-ramp merging
  • option framework

Cite this