Weighted Bootstrapped DQN: Efficient Exploration via Uncertainty Quantification

Uncertainty quantification is an essential method for sample-efficient deep reinforcement learning. There is a growing literature on uncertainty-based deep reinforcement learning algorithms, but many of the previous approaches failed to capture different sources of uncertainty. We highlight why this can be a crucial shortcoming for sample-efficient algorithms and provide a sophisticated analysis of the uncertainty in the interaction between agent and environment. Based on that, we propose Weighted Bootstrapped DQN, an exploration-efficient method that combines network ensembles and variance weighting. We use aleatoric uncertainty estimation together with epistemic uncertainty to improve the exploration ability of the algorithm. We prove that our new approach has a significant improvement in sample efficiency on different gym tasks, even compared with the previous state-of-the-art approaches.