ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks

Yan Yang; Dongxu Li; Haoning Wu; Bei Chen; Liu Liu; Liyuan Pan; Junnan Li

doi:10.18653/v1/2025.findings-acl.568

ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks

Yan Yang
, Dongxu Li^*
, Haoning Wu
, Bei Chen
, Liu Liu
, Liyuan Pan^*
, Junnan Li

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Solving expert-level multimodal tasks is a key milestone in general intelligence. As the capabilities of multimodal large language models (MLLMs) continue to evolve, evaluation of frontier multimodal intelligence becomes necessary yet challenging. In this work, we introduce ProBench, a benchmark of open-ended user queries encapsulating professional expertise and advanced reasoning. ProBench consists of 4,000 high-quality samples independently collected from professionals based on their productivity demands. It spans across 10 fields and 56 sub-fields, including science, arts, humanities, coding, mathematics, and creative writing. Experimentally, we evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal that although the best open-source models rival the proprietary ones, they all face significant challenges in visual perception, textual understanding, domain knowledge, and advanced reasoning.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	ACL 2025
Editors	Wanxiang Che, Joyce Nabende, Ekaterina Shutova, Mohammad Taher Pilehvar
Publisher	Association for Computational Linguistics (ACL)
Pages	10883-10892
Number of pages	10
ISBN (Electronic)	9798891762565
DOIs	https://doi.org/10.18653/v1/2025.findings-acl.568
Publication status	Published - 2025
Externally published	Yes
Event	63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025 - Vienna, Austria Duration: 27 Jul 2025 → 1 Aug 2025

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025
Country/Territory	Austria
City	Vienna
Period	27/07/25 → 1/08/25

Access to Document

10.18653/v1/2025.findings-acl.568

Cite this

Yang, Y., Li, D., Wu, H., Chen, B., Liu, L., Pan, L., & Li, J. (2025). ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks. In W. Che, J. Nabende, E. Shutova, & M. T. Pilehvar (Eds.), Findings of the Association for Computational Linguistics: ACL 2025 (pp. 10883-10892). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2025.findings-acl.568

Yang, Yan ; Li, Dongxu ; Wu, Haoning et al. / ProBench : Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks. Findings of the Association for Computational Linguistics: ACL 2025. editor / Wanxiang Che ; Joyce Nabende ; Ekaterina Shutova ; Mohammad Taher Pilehvar. Association for Computational Linguistics (ACL), 2025. pp. 10883-10892 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{66d808f799a4417a8cd8a302039bb039,

title = "ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks",

abstract = "Solving expert-level multimodal tasks is a key milestone in general intelligence. As the capabilities of multimodal large language models (MLLMs) continue to evolve, evaluation of frontier multimodal intelligence becomes necessary yet challenging. In this work, we introduce ProBench, a benchmark of open-ended user queries encapsulating professional expertise and advanced reasoning. ProBench consists of 4,000 high-quality samples independently collected from professionals based on their productivity demands. It spans across 10 fields and 56 sub-fields, including science, arts, humanities, coding, mathematics, and creative writing. Experimentally, we evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal that although the best open-source models rival the proprietary ones, they all face significant challenges in visual perception, textual understanding, domain knowledge, and advanced reasoning.",

author = "Yan Yang and Dongxu Li and Haoning Wu and Bei Chen and Liu Liu and Liyuan Pan and Junnan Li",

note = "Publisher Copyright: {\textcopyright} 2025 Association for Computational Linguistics.; 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025 ; Conference date: 27-07-2025 Through 01-08-2025",

year = "2025",

doi = "10.18653/v1/2025.findings-acl.568",

language = "English",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "10883--10892",

editor = "Wanxiang Che and Joyce Nabende and Ekaterina Shutova and Pilehvar, \{Mohammad Taher\}",

booktitle = "Findings of the Association for Computational Linguistics",

address = "United States",

}

Yang, Y, Li, D, Wu, H, Chen, B, Liu, L, Pan, L & Li, J 2025, ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks. in W Che, J Nabende, E Shutova & MT Pilehvar (eds), Findings of the Association for Computational Linguistics: ACL 2025. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 10883-10892, 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, 27/07/25. https://doi.org/10.18653/v1/2025.findings-acl.568

ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks. / Yang, Yan; Li, Dongxu; Wu, Haoning et al.
Findings of the Association for Computational Linguistics: ACL 2025. ed. / Wanxiang Che; Joyce Nabende; Ekaterina Shutova; Mohammad Taher Pilehvar. Association for Computational Linguistics (ACL), 2025. p. 10883-10892 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - ProBench

T2 - 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025

AU - Yang, Yan

AU - Li, Dongxu

AU - Wu, Haoning

AU - Chen, Bei

AU - Liu, Liu

AU - Pan, Liyuan

AU - Li, Junnan

PY - 2025

Y1 - 2025

N2 - Solving expert-level multimodal tasks is a key milestone in general intelligence. As the capabilities of multimodal large language models (MLLMs) continue to evolve, evaluation of frontier multimodal intelligence becomes necessary yet challenging. In this work, we introduce ProBench, a benchmark of open-ended user queries encapsulating professional expertise and advanced reasoning. ProBench consists of 4,000 high-quality samples independently collected from professionals based on their productivity demands. It spans across 10 fields and 56 sub-fields, including science, arts, humanities, coding, mathematics, and creative writing. Experimentally, we evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal that although the best open-source models rival the proprietary ones, they all face significant challenges in visual perception, textual understanding, domain knowledge, and advanced reasoning.

AB - Solving expert-level multimodal tasks is a key milestone in general intelligence. As the capabilities of multimodal large language models (MLLMs) continue to evolve, evaluation of frontier multimodal intelligence becomes necessary yet challenging. In this work, we introduce ProBench, a benchmark of open-ended user queries encapsulating professional expertise and advanced reasoning. ProBench consists of 4,000 high-quality samples independently collected from professionals based on their productivity demands. It spans across 10 fields and 56 sub-fields, including science, arts, humanities, coding, mathematics, and creative writing. Experimentally, we evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal that although the best open-source models rival the proprietary ones, they all face significant challenges in visual perception, textual understanding, domain knowledge, and advanced reasoning.

UR - https://www.scopus.com/pages/publications/105028623761

U2 - 10.18653/v1/2025.findings-acl.568

DO - 10.18653/v1/2025.findings-acl.568

M3 - Conference contribution

AN - SCOPUS:105028623761

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 10883

EP - 10892

BT - Findings of the Association for Computational Linguistics

A2 - Che, Wanxiang

A2 - Nabende, Joyce

A2 - Shutova, Ekaterina

A2 - Pilehvar, Mohammad Taher

PB - Association for Computational Linguistics (ACL)

Y2 - 27 July 2025 through 1 August 2025

ER -

Yang Y, Li D, Wu H, Chen B, Liu L, Pan L et al. ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks. In Che W, Nabende J, Shutova E, Pilehvar MT, editors, Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics (ACL). 2025. p. 10883-10892. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2025.findings-acl.568

ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this