Enhancing Curiosity-driven Reinforcement Learning through Historical State Information for Long-term Exploration

Curiosity-driven reinforcement learning is proposed to solve the sparse extrinsic reward problem by providing intrinsic rewards and achieves promising results. However, most curiosity algorithms do not acquire sufficient states knowledge and can only process local information, which would suffer from long time horizon exploration problem. In this paper, we propose a curiosity-driven exploration strategy that can continuously incentivize the curiosity of agents to address long-term exploration. In particular, we introduce a consistent exploration that provides agents with moderate knowledge, i.e., a certain amount of historical information by maintaining a finite-length state feature pool to stimulate agent's curiosity, and gives agents a novel direction of exploration. To balance the consistency and diversity of exploration directions, we propose a high-freedom exploration module that allows agents to acquire important items, such as keys to open doors, by adding actions to RND. We compared the proposed method with some recent advanced algorithms in partial Atari 2600 games environments. Experimental results demonstrate the superior performance of our method in sparse environments, especially in exploratory task with a 50% improvement in score compared to the baseline algorithm RND.