TY - GEN
T1 - Mitigating Sycophancy in Multimodal Chart Understanding via Vision-Grounded Verification
AU - Wang, Xiaolong
AU - Lin, Zhiwei
AU - Liu, Tai
AU - Li, Qiang
AU - Sun, Jian
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.
PY - 2026
Y1 - 2026
N2 - Multimodal large language models (MLLMs) have become the dominant paradigm for chart understanding, yet they suffer from a critical vulnerability: sycophancy. When users embed false premises in queries about charts, models override visual evidence to comply with the user’s misconception. Existing mitigations incur a severe safety-utility trade-off: suppressing sycophantic compliance inevitably triggers over-refusal of valid queries, degrading general performance. To address this challenge, we introduce Re-Check, a training-free inference framework that follows a “Verify-then-Answer” workflow: it first decomposes user queries into atomic claims, then verifies each claim against the chart image, and finally routes the generation through an adaptive three-way mechanism. To ensure verification reliability, we propose the Contrastive Visual Dependency Score (CVDS), an information-theoretic metric that measures the KL divergence between the model’s predictions with and without visual input, filtering out textual-bias-driven misjudgments. Through extensive experiments on diverse benchmarks, Re-Check improves overall accuracy by 15.01% (from 57.49% to 72.50% with Qwen3-VL-8B) on sycophancy scenarios, while maintaining competitive performance on general benchmarks such as ChartQAPro. These results demonstrate a superior safety-utility trade-off. Code is available at: https://github.com/X1Wang/ReCheck-code.
AB - Multimodal large language models (MLLMs) have become the dominant paradigm for chart understanding, yet they suffer from a critical vulnerability: sycophancy. When users embed false premises in queries about charts, models override visual evidence to comply with the user’s misconception. Existing mitigations incur a severe safety-utility trade-off: suppressing sycophantic compliance inevitably triggers over-refusal of valid queries, degrading general performance. To address this challenge, we introduce Re-Check, a training-free inference framework that follows a “Verify-then-Answer” workflow: it first decomposes user queries into atomic claims, then verifies each claim against the chart image, and finally routes the generation through an adaptive three-way mechanism. To ensure verification reliability, we propose the Contrastive Visual Dependency Score (CVDS), an information-theoretic metric that measures the KL divergence between the model’s predictions with and without visual input, filtering out textual-bias-driven misjudgments. Through extensive experiments on diverse benchmarks, Re-Check improves overall accuracy by 15.01% (from 57.49% to 72.50% with Qwen3-VL-8B) on sycophancy scenarios, while maintaining competitive performance on general benchmarks such as ChartQAPro. These results demonstrate a superior safety-utility trade-off. Code is available at: https://github.com/X1Wang/ReCheck-code.
KW - Chart Understanding
KW - Hallucination
KW - Multimodal Large Language Models
KW - Sycophancy Mitigation
KW - Vision-Grounded Verification
UR - https://www.scopus.com/pages/publications/105054071586
U2 - 10.1007/978-3-032-37271-0_26
DO - 10.1007/978-3-032-37271-0_26
M3 - Conference contribution
AN - SCOPUS:105054071586
SN - 9783032372703
T3 - Lecture Notes in Computer Science
SP - 477
EP - 493
BT - Computer Vision – ECCV 2026 - 19th European Conference, 2026, Proceedings
A2 - Favaro, Paolo
A2 - Kukelova, Zuzana
A2 - Maki, Atsuto
A2 - Rohrbach, Anna
A2 - Schindler, Konrad
A2 - Tombari, Federico
PB - Springer Science and Business Media Deutschland GmbH
T2 - 19th European Conference on Computer Vision, ECCV 2026
Y2 - 8 September 2026 through 12 September 2026
ER -