TY - GEN
T1 - MTAVG-Bench
T2 - 64th Annual Meeting of the Association for Computational Linguistics, ACL 2026
AU - Zhou, Yang Hao
AU - Li, Haitian
AU - Lin, Rexar
AU - Huang, Heyan
AU - Zhou, Jinxing
AU - Yuan, Changsen
AU - Lan, Tian
AU - Zhou, Ziqin
AU - Li, Yudong
AU - Xu, Jiajun
AU - Liao, Jingyun
AU - Cheng, Yiming
AU - Chen, Xuefeng
AU - Mao, Xian Ling
AU - Feng, Yousheng
N1 - Publisher Copyright:
© 2026 Association for Computational Linguistics.
PY - 2026
Y1 - 2026
N2 - Recent advances in text-to-audio-video (T2AV) generation have enabled models to synthesize audio-visual videos with multi-participant dialogues. However, existing evaluation benchmarks remain largely designed for human-recorded videos or single-speaker settings. As a result, structural failures in generated multitalker dialogue videos, such as identity drift, unnatural turn transitions, and audio-visual misalignment, cannot be effectively diagnosed. To address this issue, we introduce MTAVG-Bench, a failure-driven diagnostic benchmark for multi-talker dialogue-centric audio-video generation. MTAVG-Bench is built via a semi-automatic pipeline, where 1.8k videos are generated using mainstream T2AV models with carefully designed prompts, yielding 2.4k manually annotated QA pairs for fine-grained failure diagnosis. The benchmark evaluates multi-speaker dialogue generation at four levels: audio-visual signal fidelity, temporal attribute consistency, social interaction, and cinematic expression. Built on a hierarchical failure taxonomy and a targeted QA protocol, MTAVG-Bench is primarily designed to evaluate whether proprietary and open-source omni-models can reliably identify failure modes in multi-speaker T2AV outputs. We benchmark 12 proprietary and open-source omni-models on MTAVG-Bench, with Gemini 3 Pro achieving the strongest overall performance, while leading open-source models remain competitive in signal fidelity and consistency. Overall, MTAVG-Bench enables fine-grained failure analysis for rigorous model comparison and targeted video generation refinement.
AB - Recent advances in text-to-audio-video (T2AV) generation have enabled models to synthesize audio-visual videos with multi-participant dialogues. However, existing evaluation benchmarks remain largely designed for human-recorded videos or single-speaker settings. As a result, structural failures in generated multitalker dialogue videos, such as identity drift, unnatural turn transitions, and audio-visual misalignment, cannot be effectively diagnosed. To address this issue, we introduce MTAVG-Bench, a failure-driven diagnostic benchmark for multi-talker dialogue-centric audio-video generation. MTAVG-Bench is built via a semi-automatic pipeline, where 1.8k videos are generated using mainstream T2AV models with carefully designed prompts, yielding 2.4k manually annotated QA pairs for fine-grained failure diagnosis. The benchmark evaluates multi-speaker dialogue generation at four levels: audio-visual signal fidelity, temporal attribute consistency, social interaction, and cinematic expression. Built on a hierarchical failure taxonomy and a targeted QA protocol, MTAVG-Bench is primarily designed to evaluate whether proprietary and open-source omni-models can reliably identify failure modes in multi-speaker T2AV outputs. We benchmark 12 proprietary and open-source omni-models on MTAVG-Bench, with Gemini 3 Pro achieving the strongest overall performance, while leading open-source models remain competitive in signal fidelity and consistency. Overall, MTAVG-Bench enables fine-grained failure analysis for rigorous model comparison and targeted video generation refinement.
UR - https://www.scopus.com/pages/publications/105053635140
U2 - 10.18653/v1/2026.acl-long.913
DO - 10.18653/v1/2026.acl-long.913
M3 - Conference contribution
AN - SCOPUS:105053635140
T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics
SP - 19906
EP - 19942
BT - Long Papers
A2 - Liakata, Maria
A2 - Moreira, Viviane P.
A2 - Zhang, Jiajun
A2 - Jurgens, David
PB - Association for Computational Linguistics (ACL)
Y2 - 2 July 2026 through 7 July 2026
ER -