Textual Grounding for Open-Vocabulary Visual Information Extraction in Layout-Diversified Documents

Mengjun Cheng, Chengquan Zhang, Chang Liu*, Yuke Li, Bohan Li, Kun Yao, Xiawu Zheng, Rongrong Ji, Jie Chen

*此作品的通讯作者

科研成果: 书/报告/会议事项章节会议稿件同行评审

摘要

Current methodologies have achieved notable success in the closed-set visual information extraction (VIE) task, while the exploration into open-vocabulary settings is comparatively underdeveloped, which is practical for individual users in terms of inferring information across documents of diverse types. Existing proposal solutions, including named entity recognition methods and large language model-based methods, fall short in processing the unlimited range of open-vocabulary keys and missing explicit layout modeling. This paper introduces a novel method for tackling the given challenge by transforming the process of categorizing text tokens into a task of locating regions based on given queries also called textual grounding. Particularly, we take this a step further by pairing open-vocabulary key language embedding with corresponding grounded text visual embedding. We design a document-tailored grounding framework by incorporating layout-aware context learning and document-tailored two-stage pre-training, which significantly improves the model’s understanding of documents. Our method outperforms current proposal solutions on the SVRD benchmark for the open-vocabulary VIE task, offering lower costs and faster inference speed. Specifically, our method infers 20× faster than the QwenVL model and achieves an improvement of 24.3% in the F-score metric.

源语言英语
主期刊名Computer Vision – ECCV 2024 - 18th European Conference, Proceedings
编辑Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
出版商Springer Science and Business Media Deutschland GmbH
474-491
页数18
ISBN(印刷版)9783031729942
DOI
出版状态已出版 - 2025
已对外发布
活动18th European Conference on Computer Vision, ECCV 2024 - Milan, 意大利
期限: 29 9月 20244 10月 2024

出版系列

姓名Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
15103 LNCS
ISSN(印刷版)0302-9743
ISSN(电子版)1611-3349

会议

会议18th European Conference on Computer Vision, ECCV 2024
国家/地区意大利
Milan
时期29/09/244/10/24

指纹

探究 'Textual Grounding for Open-Vocabulary Visual Information Extraction in Layout-Diversified Documents' 的科研主题。它们共同构成独一无二的指纹。

引用此

Cheng, M., Zhang, C., Liu, C., Li, Y., Li, B., Yao, K., Zheng, X., Ji, R., & Chen, J. (2025). Textual Grounding for Open-Vocabulary Visual Information Extraction in Layout-Diversified Documents. 在 A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (编辑), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings (页码 474-491). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); 卷 15103 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-72995-9_27