跳到主要导航 跳到搜索 跳到主要内容

Unstructured Data Analysis Using LLMs: A Comprehensive Benchmark

  • Beijing Institute of Technology
  • University of Arizona
  • Hunan University

科研成果: 期刊稿件 › 会议文章 › 同行评审

摘要

The explosion of unstructured data has immense analytical value. By leveraging large language models (LLMs) to extract table-like attributes from unstructured data, researchers are building LLM-powered systems that analyze documents as if querying a database. These unstructured data analysis (UDA) systems differ widely in query interfaces, optimization, and operators, making it unclear which works best in which scenario. However, no benchmark currently offers high-quality, large-scale, diverse datasets and rich query workloads to rigorously evaluate them. We present Bench-U, a comprehensive UDA benchmark that addresses this need. We curate 6 datasets from different domains and manually construct a relational database view for each using 30 graduate students. These relational databases serve as ground truth to evaluate any UDA system, regardless of its interface. We further design diverse queries over the database schema that evaluate various analytical operators with different selectivities and complexities. Using this benchmark, we conduct an in-depth analysis of key UDA components—query interface, optimization, operator design, and data processing—and run exhaustive experiments to evaluate systems and techniques along these dimensions. Our main contributions are: (1) a comprehensive benchmark for rigorous UDA evaluation, and (2) a deeper understanding of the strengths and limitations of current systems, paving the way for future work in UDA.

源语言英语
页(从-至)2398-2410
页数13
期刊Proceedings of the VLDB Endowment
卷19
期9
DOI
出版状态已出版 - 2026
活动52nd International Conference on Very Large Data Bases, VLDB 2026 - Boston, 美国
期限: 31 8月 2026 → 4 9月 2026

学术指纹

探究 'Unstructured Data Analysis Using LLMs: A Comprehensive Benchmark' 的科研主题。它们共同构成独一无二的学术指纹。

引用此