TY - JOUR
T1 - Unstructured Data Analysis Using LLMs
T2 - 52nd International Conference on Very Large Data Bases, VLDB 2026
AU - Deng, Qiyan
AU - Li, Jianhui
AU - Chai, Chengliang
AU - Yuan, Ye
AU - Liu, Jinqi
AU - She, Junzhi
AU - Jin, Kaisen
AU - Sun, Zhaoze
AU - Deng, Yuhao
AU - Yuan, Jia
AU - Wang, Yuping
AU - Zhou, Xu
AU - Wang, Guoren
AU - Cao, Lei
N1 - Publisher Copyright:
© 2026, VLDB Endowment. All rights reserved.
PY - 2026
Y1 - 2026
N2 - The explosion of unstructured data has immense analytical value. By leveraging large language models (LLMs) to extract table-like attributes from unstructured data, researchers are building LLM-powered systems that analyze documents as if querying a database. These unstructured data analysis (UDA) systems differ widely in query interfaces, optimization, and operators, making it unclear which works best in which scenario. However, no benchmark currently offers high-quality, large-scale, diverse datasets and rich query workloads to rigorously evaluate them. We present Bench-U, a comprehensive UDA benchmark that addresses this need. We curate 6 datasets from different domains and manually construct a relational database view for each using 30 graduate students. These relational databases serve as ground truth to evaluate any UDA system, regardless of its interface. We further design diverse queries over the database schema that evaluate various analytical operators with different selectivities and complexities. Using this benchmark, we conduct an in-depth analysis of key UDA components—query interface, optimization, operator design, and data processing—and run exhaustive experiments to evaluate systems and techniques along these dimensions. Our main contributions are: (1) a comprehensive benchmark for rigorous UDA evaluation, and (2) a deeper understanding of the strengths and limitations of current systems, paving the way for future work in UDA.
AB - The explosion of unstructured data has immense analytical value. By leveraging large language models (LLMs) to extract table-like attributes from unstructured data, researchers are building LLM-powered systems that analyze documents as if querying a database. These unstructured data analysis (UDA) systems differ widely in query interfaces, optimization, and operators, making it unclear which works best in which scenario. However, no benchmark currently offers high-quality, large-scale, diverse datasets and rich query workloads to rigorously evaluate them. We present Bench-U, a comprehensive UDA benchmark that addresses this need. We curate 6 datasets from different domains and manually construct a relational database view for each using 30 graduate students. These relational databases serve as ground truth to evaluate any UDA system, regardless of its interface. We further design diverse queries over the database schema that evaluate various analytical operators with different selectivities and complexities. Using this benchmark, we conduct an in-depth analysis of key UDA components—query interface, optimization, operator design, and data processing—and run exhaustive experiments to evaluate systems and techniques along these dimensions. Our main contributions are: (1) a comprehensive benchmark for rigorous UDA evaluation, and (2) a deeper understanding of the strengths and limitations of current systems, paving the way for future work in UDA.
UR - https://www.scopus.com/pages/publications/105049181026
U2 - 10.14778/3819518.3819559
DO - 10.14778/3819518.3819559
M3 - Conference article
AN - SCOPUS:105049181026
SN - 2150-8097
VL - 19
SP - 2398
EP - 2410
JO - Proceedings of the VLDB Endowment
JF - Proceedings of the VLDB Endowment
IS - 9
Y2 - 31 August 2026 through 4 September 2026
ER -