Optical Flow Enhancement and Effect Research in Action Recognition

The accuracy of video-based action recognition depends largely on the extraction and utilization of optical flow, especially in two-stream networks. The original intention of the introduction of optical flow is to use the time information contained in video, however, the subsequent work shows that optical flow is useful for action recognition because it is invariant to appearance. In this article, we study and discuss this point of view, and propose optical flow enhancement algorithms to improve action recognition accuracy. Our enhancement algorithms improve the invariance to appearance of the representation in optical flow without losing time information, and every action recognition network with optical flow can benefit from our algorithms. We conduct a series of experiments to validate the influence of the proposed algorithms with TSN in terms of several datasets and optical flow calculation methods. As a result, we prove that first order differential algorithms are effective, TSN with our enhancement module significantly outperform original network. Based on these experiments, we also verify the importance of invariance to appearance in optical flow, and provide a reference for the follow-up study of improving action recognition accuracy.