An Improved Off-Policy Actor-Critic Algorithm with Historical Behaviors Reusing for Robotic Control

When the robot uses reinforcement learning (RL) to learn behavior policy, the requirement of RL algorithm is that it can use limited interactive data to learn the relatively optimal policy model. In this paper, we present an off-policy actor-critic deep RL algorithm based on maximum entropy RL framework. In policy improvement step, an off-policy likelihood ratio policy gradient method is derived, where the actions are sampled simultaneously from the current policy model and the experience replay buffer according to the sampled states. This method makes full use of the past experience. Moreover, we design an unified critic network, which can simultaneously approximate the state-value and action-value functions. On a range of continuous control benchmarks, the results show that our method outperforms the state-of-the-art soft actor-critic (SAC) algorithm in stability and asymptotic performance.