Enhancing multimodal interaction in head-mounted displays via integrating on-arm touch and arm-pointing

The human body serves as an always-available interaction interface, with the arm being a commonly used medium, especially in head-mounted displays (HMDs). Leveraging the body’s inherent perceptual capabilities, users can perform stable and accurate arm-based interactions even without visual feedback (eyes-free conditions). Arm-based input can be realized through touch gestures performed by the opposite hand (on-arm touch) or via arm orientation for directional input (arm-pointing). However, existing approaches often rely on visual guidance and unimodal input, limiting their flexibility and effectiveness. Therefore, we developed a prototype that enables multimodal interaction on the arm with eyes-free, supporting both touch-based and pointing-based inputs. We conducted a user study to investigate input performance when using dominant versus non-dominant hands for on-arm touch input and compared three localization strategies. To further understand pointing-based input without visual guidance, we designed a Fitts’ law experiment under a decoupled arm-pointing and display-alignment setting, examining how different arm placements affect performance. Finally, we conducted a comparative study to evaluate the efficiency of multimodal input methods that combine on-arm touch and pointing input. Results show that our method significantly reduces interaction time and improves accuracy compared with the traditional method. This work presents the first implementation of a multimodal arm-based interaction method, offering valuable insights for designing efficient and intuitive interfaces in future AR/VR applications.