Skip to main navigation Skip to search Skip to main content

Unsupervised Interpretable Human Intent Discovery in Teleoperation via Bayesian Nonparametric Learning and Multi-Modal LLMs

  • Beijing Institute of Technology
  • Beijing Institute of Technology (Zhuhai)

Research output: Contribution to journal › Article › peer-review

Abstract

Human intent recognition is crucial for effective robotic teleoperation, yet existing methods rely on a predefined set of intentions, a requirement that is manually intractable in complex open-world scenarios. We propose an unsupervised framework that integrates Bayesian nonparametric learning with multi-modal Large Language Models (LLMs) to autonomously discover and interpret human intentions. We first employ a hierarchical Dirichlet process hidden semi-Markov model (HDP-HSMM) to adaptively segment kinematic sequences into motion primitives, reducing the load on downstream models. Each segment is represented by multi-modal features extracted from front-facing video, robot kinematics, and eye fixations. We then develop a reasoning architecture based on zero-shot Chain-of-Thought (CoT) prompting, where a Vision-Language Model (VLM) translates each segment into a semantic description, and an LLM clusters these descriptions to reveal intentions and generate human-readable interpretations. Experimental results on real-world data show that our method can uncover meaningful intentions highly consistent with expert judgments, outperforming baselines in intra-intent consistency by achieving the lowest semantic outlier rate of 8.5%.

Original languageEnglish
Pages (from-to)12512-12519
Number of pages8
JournalIEEE Robotics and Automation Letters
Volume11
Issue number11
DOIs
Publication statusPublished - 1 Nov 2026
Externally publishedYes

Keywords

  • Bayesian nonparametric learning
  • Human intent discovery
  • large language models
  • robotic teleoperation

Fingerprint

Dive into the research topics of 'Unsupervised Interpretable Human Intent Discovery in Teleoperation via Bayesian Nonparametric Learning and Multi-Modal LLMs'. Together they form a unique fingerprint.

Cite this