Differential High Order Control Barrier Function-Based Safe Reinforcement Learning

Safe reinforcement learning (RL) aims to learn policy while also ensuring the safety constraints. An increasingly common approach is to design a safety filter based on control barrier function (CBF) or high order control barrier function (HOCBF) for the RL policy. A quadratic programming (QP) is then formulated and solved to modify the RL policy, enabling safe exploration. However, directly integrating the safety filter with RL presents two challenges: (1) the conservativeness of safe policy, and (2) potential infeasibility of the QP under bounded input constraints. These issues limit the performance of safe RL. In this letter, we introduce a differential HOCBF constraint by incorporating neural network-based penalty functions into HOCBF. Furthermore, we propose a differential HOCBF-based safe RL framework in which the penalty functions and RL policy are trained concurrently. To address conservativeness, we train penalty functions to maximize long-term rewards while preventing abrupt changes in safe action, thereby achieving ideal performance. To ensure the feasibility of the formulated QP under bounded input constraints, we calculate a set for penalty functions and prove that the feasibility is guaranteed if the learned penalty functions remain within the set. Finally, we verify the effectiveness of the proposed framework on the wheeled mobile robot navigation and obstacle avoidance task.