KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for visual commonsense reasoning[Formula presented]

Dandan Song*, Siyi Ma, Zhanchen Sun, Sicheng Yang, Lejian Liao

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

26 Citations (Scopus)

Abstract

Reasoning is a critical ability towards complete visual understanding. To develop machine with cognition-level visual understanding and reasoning abilities, the visual commonsense reasoning (VCR) task has been introduced. In VCR, given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. The methods adopting the powerful BERT model as the backbone for learning joint representation of image content and natural language have shown promising improvements on VCR. However, none of the existing methods have utilized commonsense knowledge in visual commonsense reasoning, which we believe will be greatly helpful in this task. Therefore, we incorporate commonsense knowledge into the cross-modal BERT, and propose a novel Knowledge Enhanced Visual-and-Linguistic BERT (KVL-BERT for short) model. Besides taking visual and linguistic contents as input, external commonsense knowledge is integrated into the multi-layer Transformer. In order to preserve the structural information and semantic representation of the original sentence, we propose an algorithm called RMGSR (Relative-position-embedding and Mask-self-attention Guided Semantic Representations). Compared to other task-specific models and general task-agnostic pre-training models, our KVL-BERT outperforms them.

Original languageEnglish
Article number107408
JournalKnowledge-Based Systems
Volume230
DOIs
Publication statusPublished - 27 Oct 2021

Keywords

  • Commonsense knowledge integration
  • Multimodal BERT
  • Visual commonsense reasoning

Fingerprint

Dive into the research topics of 'KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for visual commonsense reasoning[Formula presented]'. Together they form a unique fingerprint.

Cite this