AcouWrite: Acoustic-Based Handwriting Recognition on Smartphones

Off-screen handwriting recognition enriches the handwriting interaction paradigm for mobile devices. However, the existing approaches are only applicable to the specific environment and equipment conditions. In this article, we propose AcouWrite, a general, scalable and real-time handwriting recognition system based on active acoustic sensing. In detail, AcouWrite relies on active acoustic sensing using only a pair of microphones and speakers on the smartphone to capture real-time handwriting input. Particularly, we extract the short-time dCIR (st-dCIR) to monitor the changes in the acoustic transmission channel resulting from finger movement. Technically, we use a CNN-GRU classifier to complete the recognition task in AcouWrite. Moreover, we use data augmentation and spelling error correction methods to improve AcouWrite's robustness. To improve the generalization of our AcouWrite for new characters, we incorporate the transfer learning module into our AcouWrite. In various real-world environments, experiments demonstrate that AcouWrite achieves a mean recognition accuracy of 97.62%, a word accuracy (WA) of 96.4% and a character error rate (CER) of 1.5% for 100 common words, and an average response time of 94 milliseconds.