Skeleton Capsule Net: An Efficient Network for Action Recognition

Capsule network is a new type of deep learning method to improve the CNN module. Though it has performed quite well on classifying the MNIST dataset, there are few applications in other fields. Thus in this paper, we apply the capsule network on skeleton-based classification and propose a framework to explore the potential of it. Since the bottom layer of the capsule network is still based on convolution operation, we feed heatmap as well as raw skeleton data and reach good performance on convolution-based action recognition. Most researches take spatial and temporal features into consideration and they do help to recognition accuracy. We propose two different encapsulations to extract the spatial and temporal features of skeleton sequences. We perform our experiments on UT-Kinect and a portion of NTU RGB+D dataset, and we achieve best 87% accuracy on the NTU RGB+D dataset. We also find that the capsule network is suitable for the coarse-grained classification tasks. In a conclusion, not only the characteristics of capsule network are proved, but also an efficient method to recognize human action is realized.