Freeze-CD: Alleviating Hallucination of Large Language Models via Contrastive Decoding with Local Freezing Training

Large Language Models (LLMs) have demonstrated remarkable capabilities across various natural language processing tasks, but they are prone to generating hallucinated contents which are inconsistent with the facts. Previous research has explored contrastive decoding between an original model and an amateur model induced by hallucination to address this issue and achieve the excellent results. However, such approach may inadvertently disrupt the output distribution of the original model due to its rough contrast and direct calculation of logits, leading to poor performance in some cases. In this paper, we propose the Freeze-CD, a novel method that mitigates hallucination by introducing an extra set of contrastive decoding between the original model and an amateur model constructed by hallucination corpus with local freezing training to enhance some local information of the original model during the decoding process. Experimental results on two publicly available benchmarks demonstrate that our approach can significantly curtail the hallucination, presenting a refined solution to improve the reliability of LLMs.