Constructing LTL-based Reward Machines for Reinforcement Learning via Bayesian Approach

Reward Machines (RMs) can tackle sparse reward challenges in Reinforcement Learning, thus leading to more efficient training. However, their practical application is hindered by the requirement of extensive expert knowledge for manual design. Here, we introduce a novel Bayesian method to construct RMs without hand-crafting. Embed with linear temporal logic, our method discovers temporal specifications from complex environment tasks, and the specifications are subsequently translated into RMs. The constructed RMs enable agents to perform more goal-directed actions and clearly understand task progression when facing sparse rewards. Experimental results demonstrate that our method quickly generates highly interpretable and robust RMs compared to existing approaches, even from noisy data in partially observable environments.