Skip to main navigation Skip to search Skip to main content

PEAP: Proactive Embodied Action Sequence Planning with Joint Understanding of Vision and Audio Perception

  • Tianwei Lan
  • , Jiaqi Wu
  • , Zeming Liu
  • , Zhaoxin Fan
  • , Haifeng Wang
  • , Yuhang Guo*
  • *Corresponding author for this work
  • Beijing Institute of Technology
  • Beihang University
  • Baidu Inc

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Embodied Action Sequence Planning focuses on the capability of embodied agents to implement action planning via environmental perception. This technology enables diverse intelligent assistance for real-world scenarios such as home and office environments. To address the limitations of existing embodied agents in meeting the requirement for proactivity and achieving joint understanding of visual and audio information, this study investigates the ability of embodied agents to proactively provide assistance through action sequence planning based on joint understanding of vision and audio perception without explicit human instructions. Correspondingly, we propose PEAP, the first multimodal proactive embodied action sequence planning dataset. We evaluate the performance of multiple Large Language Models on the PEAP dataset. The results demonstrate that these models still exhibit significant deficiencies on this task particularly lacking accurate environmental perception capabilities. Furthermore, ablation experiment and replacement experiment further corroborate that the joint understanding of multimodal information can significantly improve the models' performance on proactive embodied action sequence planning task. Our dataset and code are publicly available.

Original languageEnglish
Title of host publicationLong Papers
EditorsMaria Liakata, Viviane P. Moreira, Jiajun Zhang, David Jurgens
PublisherAssociation for Computational Linguistics (ACL)
Pages23118-23138
Number of pages21
ISBN (Electronic)9798891763906
DOIs
Publication statusPublished - 2026
Externally publishedYes
Event64th Annual Meeting of the Association for Computational Linguistics, ACL 2026 - San Diego, United States
Duration: 2 Jul 2026 → 7 Jul 2026

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
Volume1
ISSN (Print)0736-587X

Conference

Conference64th Annual Meeting of the Association for Computational Linguistics, ACL 2026
Country/TerritoryUnited States
CitySan Diego
Period2/07/26 → 7/07/26

Fingerprint

Dive into the research topics of 'PEAP: Proactive Embodied Action Sequence Planning with Joint Understanding of Vision and Audio Perception'. Together they form a unique fingerprint.

Cite this