Skip to main navigation Skip to search Skip to main content

MTAVG-Bench: A Diagnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation

  • Yang Hao Zhou
  • , Haitian Li
  • , Rexar Lin
  • , Heyan Huang*
  • , Jinxing Zhou
  • , Changsen Yuan
  • , Tian Lan
  • , Ziqin Zhou
  • , Yudong Li
  • , Jiajun Xu
  • , Jingyun Liao
  • , Yiming Cheng
  • , Xuefeng Chen
  • , Xian Ling Mao
  • , Yousheng Feng
  • *Corresponding author for this work
  • Beijing Institute of Technology
  • Shanghai University
  • OpenNLP Lab
  • Beijing University of Technology
  • Adelaide University
  • Tsinghua University
  • Inkeverse Group Limited

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Recent advances in text-to-audio-video (T2AV) generation have enabled models to synthesize audio-visual videos with multi-participant dialogues. However, existing evaluation benchmarks remain largely designed for human-recorded videos or single-speaker settings. As a result, structural failures in generated multitalker dialogue videos, such as identity drift, unnatural turn transitions, and audio-visual misalignment, cannot be effectively diagnosed. To address this issue, we introduce MTAVG-Bench, a failure-driven diagnostic benchmark for multi-talker dialogue-centric audio-video generation. MTAVG-Bench is built via a semi-automatic pipeline, where 1.8k videos are generated using mainstream T2AV models with carefully designed prompts, yielding 2.4k manually annotated QA pairs for fine-grained failure diagnosis. The benchmark evaluates multi-speaker dialogue generation at four levels: audio-visual signal fidelity, temporal attribute consistency, social interaction, and cinematic expression. Built on a hierarchical failure taxonomy and a targeted QA protocol, MTAVG-Bench is primarily designed to evaluate whether proprietary and open-source omni-models can reliably identify failure modes in multi-speaker T2AV outputs. We benchmark 12 proprietary and open-source omni-models on MTAVG-Bench, with Gemini 3 Pro achieving the strongest overall performance, while leading open-source models remain competitive in signal fidelity and consistency. Overall, MTAVG-Bench enables fine-grained failure analysis for rigorous model comparison and targeted video generation refinement.

Original languageEnglish
Title of host publicationLong Papers
EditorsMaria Liakata, Viviane P. Moreira, Jiajun Zhang, David Jurgens
PublisherAssociation for Computational Linguistics (ACL)
Pages19906-19942
Number of pages37
ISBN (Electronic)9798891763906
DOIs
Publication statusPublished - 2026
Event64th Annual Meeting of the Association for Computational Linguistics, ACL 2026 - San Diego, United States
Duration: 2 Jul 2026 → 7 Jul 2026

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
Volume1
ISSN (Print)0736-587X

Conference

Conference64th Annual Meeting of the Association for Computational Linguistics, ACL 2026
Country/TerritoryUnited States
CitySan Diego
Period2/07/26 → 7/07/26

Fingerprint

Dive into the research topics of 'MTAVG-Bench: A Diagnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation'. Together they form a unique fingerprint.

Cite this