SignParser: Empowering Dual-Handed Sign Language Translation with a Single Wearable

Sign language translation (SLT) is essential for promoting communicative equity and social integration for hearing-impaired individuals. However, computer-vision-based and wireless-signal-based SLT solutions mainly involve inconvenient operation, poor portability, and susceptibility to interference. Recently, wearable-device-based methods have emerged as a potential alternative, offering services anytime and anywhere. However, these methods fall into two extremes: employing complex device combinations to achieve dual-handed SLT or opting for single-device solutions that compromise comprehensive data capture from both hands. Consequently, such a dilemma constrains the widespread adoption of wearable devices in the field of SLT. In this paper, we propose SignParser, a unique dual-handed SLT system leveraging a single IMU sensor in commercial smartwatches. SignParser is superior to other wearable-device-based approaches in i) exploiting large-scale labeled virtual IMU data to achieve generalization capability across different users, ii) enabling single-device solution for dual-handed SLT via estimating non-dominant hand IMU data, and iii) ensuring real-time, contextual-guided, and unseen sentence-adaptive SLT by a lightweight sign spotter network integrated with large language models. Extensive experiments with 27 participants show that SignParser can achieve the average word error rate of 4.8% and 8.3% for new users and unseen sentences, respectively. The excellent performance demonstrates the SignParser's effectiveness in real-world scenarios.