Human-centric multimodal data processing for human-robot collaboration: A survey

In the context of Industry 5.0, which advocates human-centric manufacturing, human-robot collaboration (HRC) has emerged as a key paradigm for enhancing production flexibility and efficiency by integrating humans’ cognitive and adaptive capabilities with robots’ high-precision execution. In complex and dynamic collaborative environments, natural interaction and effective teamwork rely heavily on multimodal data acquisition, representation, alignment, fusion, and reasoning. However, existing surveys predominantly focus on general HRC frameworks or application-oriented perspectives, while a systematic review of multimodal data processing methods and their functional roles throughout the collaboration pipeline, remains limited. To bridge this gap, we adopt a multimodal data-driven, end-to-end perspective on HRC (covering studies up to June 30, 2025). A total of 191 relevant studies are reviewed, and an integrated research framework is established encompassing human-robot relationship modeling, perception, cognition, decision-making, and application analysis. In particular, we analyze core methodologies for relationship modeling, unimodal representation learning, cross-modal alignment and fusion, as well as semantic reasoning and decision-making, and discuss their roles in environment perception, user understanding, and collaborative assistance. Finally, key challenges in human-robot relationship modeling, multimodal perception, data modeling, and reasoning are identified, and future research directions are outlined. This survey aims to provide a systematic understanding of multimodal data processing in HRC and to stimulate further research and academic discourse in this rapidly evolving field.