跳到主要导航 跳到搜索 跳到主要内容

BatchEval: Towards Human-like Text Evaluation

  • Peiwen Yuan
  • , Shaoxiong Feng
  • , Yiwei Li
  • , Xinglin Wang
  • , Boyuan Pan
  • , Heda Wang
  • , Yao Hu
  • , Kan Li*
  • *此作品的通讯作者
  • Beijing Institute of Technology
  • Xiaohongshu

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigm suffers from the following issues: (1) Sensitive to prompt design; (2) Poor resistance to context noise; (3) Inferior ensemble performance with static reference. Inspired by the fact that humans treat both criterion definition and inter sample comparison as references for evaluation, we propose BATCHEVAL, a paradigm that conducts batch-wise evaluation iteratively to alleviate the above problems. We explore variants under this paradigm and confirm the optimal settings are two stage procedure with heterogeneous batch composition strategy and decimal scoring format. Comprehensive experiments across 3 LLMs on 4 text evaluation tasks demonstrate that BATCHEVAL outperforms state-of-the-art methods by 10.5% on Pearson correlations with only 64% API cost on average. Further analyses have verified the robustness, generalization, and working mechanism of BATCHEVAL.

源语言英语
主期刊名Long Papers
编辑Lun-Wei Ku, Andre F. T. Martins, Vivek Srikumar
出版商Association for Computational Linguistics (ACL)
页15940-15958
页数19
ISBN(电子版)9798891760943
DOI
出版状态已出版 - 2024
活动62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Bangkok, 泰国
期限: 11 8月 2024 → 16 8月 2024

丛书

姓名Proceedings of the Annual Meeting of the Association for Computational Linguistics
卷1
ISSN(印刷版)0736-587X

会议

会议62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024
国家/地区泰国
市Bangkok
时期11/08/24 → 16/08/24

学术指纹

探究 'BatchEval: Towards Human-like Text Evaluation' 的科研主题。它们共同构成独一无二的学术指纹。

引用此