Continuous Arm Motion Recognition Using Two-stream Spatial-Temporal Neural Network Based on Millimeter Wave Sensor

Radar-based continuous arm motion recognition has attracted increasing interests from research community due to its huge potential to interact with machine in a longer distance. Whereas, most existing arm motion recognition work have been conducted on individual motion or without caring for the preceding and subsequent motion information in the continuous data stream. Practical operation also requires that the system must continuously detect and recognize arm motions using untruncated input radar echo streams with unknown transitions between motions. In this paper, to address these challenges, a two-stream neural network accounting for the spatial-temporal information of the radar data stream is proposed for automatic continuous arm motion recognition with a Doppler radar sensor. One stream of the network combined 1-D CNN and Bi-LSTM in a sequential manner to extract the bi-direction long-term temporal dependencies of deep column-wise spatial features. The other stream used ResNet50 to extract the global spatial features. The two are then fused to output the final class label. The comparison with the state-of-art continuous motion recognition network, 1-D CNN-Bi-LSTM, shows that our proposed network structure achieves higher recognition accuracy. Furthermore, by applying a sliding window to the data stream, our approach can handle with random length arm motion echo stream with unconstrained transitions.