Skip to main navigation Skip to search Skip to main content

Mitigating Sycophancy in Multimodal Chart Understanding via Vision-Grounded Verification

  • Xiaolong Wang
  • , Zhiwei Lin*
  • , Tai Liu
  • , Qiang Li
  • , Jian Sun*
  • *Corresponding author for this work
  • Beijing Institute of Technology
  • China Academy of Information and Communications Technology
  • Ltd.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Multimodal large language models (MLLMs) have become the dominant paradigm for chart understanding, yet they suffer from a critical vulnerability: sycophancy. When users embed false premises in queries about charts, models override visual evidence to comply with the user’s misconception. Existing mitigations incur a severe safety-utility trade-off: suppressing sycophantic compliance inevitably triggers over-refusal of valid queries, degrading general performance. To address this challenge, we introduce Re-Check, a training-free inference framework that follows a “Verify-then-Answer” workflow: it first decomposes user queries into atomic claims, then verifies each claim against the chart image, and finally routes the generation through an adaptive three-way mechanism. To ensure verification reliability, we propose the Contrastive Visual Dependency Score (CVDS), an information-theoretic metric that measures the KL divergence between the model’s predictions with and without visual input, filtering out textual-bias-driven misjudgments. Through extensive experiments on diverse benchmarks, Re-Check improves overall accuracy by 15.01% (from 57.49% to 72.50% with Qwen3-VL-8B) on sycophancy scenarios, while maintaining competitive performance on general benchmarks such as ChartQAPro. These results demonstrate a superior safety-utility trade-off. Code is available at: https://github.com/X1Wang/ReCheck-code.

Original languageEnglish
Title of host publicationComputer Vision – ECCV 2026 - 19th European Conference, 2026, Proceedings
EditorsPaolo Favaro, Zuzana Kukelova, Atsuto Maki, Anna Rohrbach, Konrad Schindler, Federico Tombari
PublisherSpringer Science and Business Media Deutschland GmbH
Pages477-493
Number of pages17
ISBN (Print)9783032372703
DOIs
Publication statusPublished - 2026
Event19th European Conference on Computer Vision, ECCV 2026 - Malmö, Sweden
Duration: 8 Sept 2026 → 12 Sept 2026

Publication series

NameLecture Notes in Computer Science
Volume17013 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference19th European Conference on Computer Vision, ECCV 2026
Country/TerritorySweden
CityMalmö
Period8/09/26 → 12/09/26

Keywords

  • Chart Understanding
  • Hallucination
  • Multimodal Large Language Models
  • Sycophancy Mitigation
  • Vision-Grounded Verification

Fingerprint

Dive into the research topics of 'Mitigating Sycophancy in Multimodal Chart Understanding via Vision-Grounded Verification'. Together they form a unique fingerprint.

Cite this