Perception-decision-execution coordination mechanism driven dynamic autonomous collaboration method for human-like collaborative robot based on multimodal large language model

With the advent of Industry 5.0, human-centric smart manufacturing is becoming a new paradigm for industrial transformation. Human-robot collaboration (HRC) is the hot topic of human-centric smart manufacturing. The emergence of large language model (LLM) provides significant opportunity for collaborative robot to promote the autonomous collaboration ability, which brings HRC into new era driven by embodied intelligence and more powerful robot. Therefore, a dynamic autonomous collaboration method inspired from looking-thinking-doing chain of human operators is proposed for human-like collaborative robot (HLCobot) in human-centric smart manufacturing based on multimodal large language model (MLLM), where perception-decision-execution coordination mechanism is constructed to appropriately distribute the abilities of MLLM in the dynamic operation chain of HRC. Firstly, a brain-inspired architecture with the integration of perception hub, decision hub, and execution hub is designed for dynamic autonomous collaboration. Secondly, the abilities of perception, decision, execution of HLCobot are realized by integrating MLLM, where the HLCobot can actively recognize the dynamic changes of HRC scenario by mimicking human operator and execute the correct motions to complete the necessary collaborative task autonomously. Additionally, a coordination mechanism among the agents of perception, decision, and execution is put forward to proceed the collaborative task smoothly. Finally, a case study of engine assembly is provided to demonstrate the effectiveness of the proposed method.