Dual-Stage Reinforcement Learning-Based Beam Tracking for Integrated Sensing and Communications in V2I Scenarios

In integrated sensing and communications (ISAC)-enabled vehicle-to-infrastructure (V2I) scenarios, vehicles are modeled as extended targets, which causes an angle mismatch between the directions of the echo signal and the communication signal, necessitating accurate beam tracking. However, the rapidly changing propagation environment in V2I scenarios, along with the use of directional beams, makes beam tracking highly challenging. To overcome the above challenges, this paper presents a dual-stage reinforcement learning (DSRL)-based beam-tracking method for ISAC-enabled V2I scenarios. Specifically, in the first stage, the vehicle of interest is modeled as an extended target, and the codebook is designed by comprehensively considering the vehicle length, the antenna spacing, the beam angle, etc. Then, considering the mobility of the vehicle, a dynamic quality value (Q)-learning-based method is proposed for coarse beam selection, which aims to guarantee that the target vehicle is covered. In the second stage, considering the angle offset between the directions of the vehicle center and the aligned beam from the first stage, the deep Q-network (DQN) algorithm is designed to solve the fine beam-tracking problem. The simulation results demonstrate that the proposed beam-tracking method outperforms existing methods in terms of the achievable rate and tracking time.