In-Context Learning Reward Guided Decoding for Controlled Text Generation

While large language models have demonstrated remarkable text generation capabilities, they often generate text with adverse or undesired attributes. Common approaches to control text generation involve refining models on data with desired properties or guiding language models decoding using an auxiliary model. However, these methods require additional training and extensive attribute-specific data. To further mitigate the training costs, we propose In-context learning Reward Guided Decoding (IRGD), a weighted decoding method that exploits the in-context learning ability of language models as an alternative to additional model fine-tuning. Specifically, IRGD utilizes ICL outputs to score the alignment reward between sequences and target attributes, subsequently modifying the sampling probabilities to favor tokens with higher reward scores. By applying ICL, IRGD adapts to different tasks by simply adjusting task descriptions and demonstration rather than fine-tuning the model. Through experiments on detoxification and sentiment control, we demonstrate the advantages of IRGD as a plug-and-play and fine-tuning-free decoding method that effectively balance attribute alignment and text quality.