Hierarchical Reinforcement Learning Combined with Motion Primitives for Automated Overtaking

This paper presents a novel hierarchical reinforcement learning (HRL) framework for automated overtaking. The proposed framework is developed based on the semi-Markov decision process (SMDP) and motion primitives (MPs) which can be applied to different overtaking phases. Unlike the high-level decision and low-level control which are usually independent with each other, the high-level decision making and low-level control are combined by defining MPs with different time intervals. As for the high-level decision making, a SMDP Q-learning algorithm is adopted to realize decision-making of MPs. Besides, a development method of MPs used in the low-level control of automated overtaking is proposed. The performance of the HRL framework is tested in the simulation environment built in a driving simulator called CARLA. The results show that the HRL framework can determine the optimal trajectory under different driving styles of the overtaken vehicle.