Tracking and Clustering Experience Replay for Balancing Sample Immediacy and Diversity

In reinforcement learning (RL), the experience replay mechanism plays an important role in collecting and sampling transitions. However, traditional replay mechanisms are limited by inefficient utilization of experience and unbalanced training between recent and past transitions. In this paper, we propose the Tracking and Clustering Experience Replay (TCER) to overcome these limitations. TCER composes of two sampling strategies: tracking sampling and clustering sampling. Tracking sampling continuously tracks and replays the latest transitions, while clustering sampling equally selects transitions among different clusters from the previous period throughout training. The combination of these two strategies ensures a balance of sample immediacy and diversity, thereby improving the performance of RL training. The experimental results on the Gymnasium tasks demonstrate the individual contribution of each strategy and validate the effectiveness of the proposed algorithm.