Emotion Recognition Based on Multi-View Body Gestures

Body gesture, a crucial component of 'body language', remains less explored to recognize emotion while face expression-based and speech-based approaches are widely investigated. In this paper, we introduce an exploratory experiment to recognize emotion using deep learning only from body gestures. 43,200 multi-view RGB videos of simplified body gestures and their neutral control groups are captured from 80 humans using Hikvision network cameras to support the experiment. A novel approach is proposed to use deep neural network fuse skeleton and RGB features only using single-modality RGB video data. Experimental results show our approach achieves substantial improvements both in individual categories and overall and is provided with stronger generalization capability as well.