ARNets: Action Recurrent Networks for Human Action Recognition

Recurrent neural networks (RNNs) are powerful and robust neural networks that capture time dynamics via cycles in the graph. Compared to Convolutional Neural Networks (CNNs), RNNs consider both the current and the historical inputs which leads them to the standard approach for practitioners to address machine learning tasks involving sequential data. Compared to the achievements in these sequential learning tasks, RNNs pale in presenting a promising performance on human action recognition. Do we miss something? In this paper, we will conduct a detailed investigation of the RNNs for the human action recognition task. Several alternative recurrent architectures are designed and compared. Even simply, we find that adding the supervision signal at the end of an RNN can provide us with a comparable if not better performance than the previous state-of-the-art. We further analyze the different combinations of modalities and the effect of the length of the sequence. Compared to CNN-based recognition architecture, RNNs focus more on the action itself. Going deeper, we also find that when modeling long and difficult actions, RNNs perform much better than CNNs. It means a complex video action dataset may be required.