Enhancing UAV Human-Machine Interaction With Multimodal Behavioral Data: A Gaze-Posture Synergistic Approach

Recent advances in human-machine interaction (HMI) for unmanned aerial vehicles (UAVs) have highlighted the limitations of traditional single-modal interfaces, which often suffer from limited adaptability and vulnerability to sensor-level anomalies. These challenges become particularly critical when environmental noise or potential adversarial interference could compromise operational safety. To address these issues, this paper proposes the Gaze-Posture Integrated UAV (GPIUAV) framework, a multimodal interaction system that fuses eye-tracking inputs with head pose measurements. The framework incorporates three core modules: a head pose estimation algorithm based on Kalman filtering, a real-time gaze-to-scene coordinate mapping method, and a multimodal fusion control scheme. Experimental evaluation in real-world UAV tasks demonstrates that GPIUAV achieves an average control accuracy of 92.5%, a mean response time of 1.106 seconds, and consistent task completion performance. Compared to manual operation, this consistency indicates that the framework reduces reliance on operator proficiency and enhances control stability. These results validate the system’s effectiveness in enabling intuitive and precise control of UAVs. Furthermore, the fusion of complementary behavioral signals offers a pathway to future improvements in operational safety and resilience through cross-modal consistency checks. The GPIUAV framework thus contributes to more reliable human-UAV collaboration in domains such as medical delivery, urban monitoring, and emergency response.