Freeze-CD: Alleviating Hallucination of Large Language Models via Contrastive Decoding with Local Freezing Training

Dingwei Chen, Shuai Wang, Zhengping Fan*, Xiping Hu, Chengming Li*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various natural language processing tasks, but they are prone to generating hallucinated contents which are inconsistent with the facts. Previous research has explored contrastive decoding between an original model and an amateur model induced by hallucination to address this issue and achieve the excellent results. However, such approach may inadvertently disrupt the output distribution of the original model due to its rough contrast and direct calculation of logits, leading to poor performance in some cases. In this paper, we propose the Freeze-CD, a novel method that mitigates hallucination by introducing an extra set of contrastive decoding between the original model and an amateur model constructed by hallucination corpus with local freezing training to enhance some local information of the original model during the decoding process. Experimental results on two publicly available benchmarks demonstrate that our approach can significantly curtail the hallucination, presenting a refined solution to improve the reliability of LLMs.

Original languageEnglish
Title of host publicationProceedings - 2024 IEEE International Conference on Smart Internet of Things, SmartIoT 2024
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages325-329
Number of pages5
ISBN (Electronic)9798350366440
DOIs
Publication statusPublished - 2024
Externally publishedYes
Event8th IEEE International Conference on Smart Internet of Things, SmartIoT 2024 - Shenzhen, China
Duration: 14 Nov 202416 Nov 2024

Publication series

NameProceedings - 2024 IEEE International Conference on Smart Internet of Things, SmartIoT 2024

Conference

Conference8th IEEE International Conference on Smart Internet of Things, SmartIoT 2024
Country/TerritoryChina
CityShenzhen
Period14/11/2416/11/24

Keywords

  • contrastive decoding
  • freezing training
  • hallucination
  • Large language model

Fingerprint

Dive into the research topics of 'Freeze-CD: Alleviating Hallucination of Large Language Models via Contrastive Decoding with Local Freezing Training'. Together they form a unique fingerprint.

Cite this