跳到主要导航 跳到搜索 跳到主要内容

ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning

  • Zhiwei Hao
  • , Jianyuan Guo
  • , Li Shen
  • , Yong Luo
  • , Han Hu*
  • , Yonggang Wen
  • *此作品的通讯作者
  • Beijing Institute of Technology
  • City University of Hong Kong
  • Sun Yat-Sen University
  • Wuhan University
  • Nanyang Technological University

科研成果: 期刊稿件 › 文章 › 同行评审

摘要

Recent advancements in multimodal fusion have witnessed the remarkable success of vision-language (VL) models, which excel in various multimodal applications such as image captioning and visual question answering. However, building VL models requires substantial hardware resources, where efficiency is restricted by two key factors: the extended input sequence of the language model with vision features demands more computational operations, and a large number of additional learnable parameters increase memory complexity. These challenges significantly restrict the broader applicability of such models. To bridge this gap, we propose ADEM-VL, an efficient vision-language method that tunes VL models based on pretrained large language models (LLMs) by adopting a parameter-free cross-attention mechanism for similarity measurements in multimodal fusion. This approach only requires embedding vision features into the language space, significantly reducing the number of trainable parameters and accelerating both training and inference speeds. To enhance representation learning in fusion module, we introduce an efficient multiscale feature generation scheme that requires only a single forward pass through the vision encoder. Moreover, we propose an adaptive fusion scheme that dynamically discards less relevant visual information for each text token based on its attention score. This ensures that the fusion process prioritizes the most pertinent visual features. With experiments on various tasks including visual question answering, image captioning, and instruction-following, we demonstrate that our framework outperforms existing approaches. Specifically, our method surpasses existing methods by an average accuracy of 0.77% on ScienceQA dataset, with reduced training and inference latency, demonstrating the superiority of our framework.

源语言英语
页(从-至)5527-5543
页数17
期刊International Journal of Computer Vision
卷133
期8
DOI
出版状态已出版 - 8月 2025
已对外发布是

学术指纹

探究 'ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning' 的科研主题。它们共同构成独一无二的学术指纹。

引用此