Observation-Time-Action Deep Stacking Strategy: Solving Partial Observability Problems with Visual Input

Reinforcement learning tasks that involve visual input continue to pose a challenge when it comes to partial observability problems. Although prior research has introduced methods such as LSTM, GTrXL, and DNC, each of these approaches possesses its own limitations. To address the problem of partial observability in a more universal context, this paper proposed the Observation-Time-Action deep stacking algorithm. First, observations, actions, and time data were combined into a tuple and then stacked into a longer sequence. Then, convolution and fully connected layers were utilized to extract relevant features from the sequence, which were then fed into the algorithm for processing. We designed a number of experiments with partial observability, corresponding to different typical scenarios in reinforcement learning. The experiment results demonstrated that the proposed method obtained a higher success rate. Moreover, we also investigated the effect of stacking frame length and different reinforcement learning elements on the algorithm. Finally, we conducted a HITL (Hardware-in-the-Loop) experiment to further verify the effectiveness of our algorithm.