An epoch-weighted privacy budget allocation framework for fine-tuning large language models

Fine-tuning large language models (LLMs) has demonstrated outstanding performance across various downstream tasks. However, fine-tuning LLMs on private training data poses significant privacy risks, as adversaries can employ various attack methods to extract sensitive information during training. Existing differential privacy frameworks for LLMs often rely on a uniform privacy budget allocation strategy, which neglects the increasing sensitivity of model parameters to noise perturbations as optimization progresses. To address these issues, we propose an epoch-weighted privacy budget allocation framework for fine-tuning LLMs (EW-FT), which incorporates epoch factors into the privacy budget allocation algorithm and utilizes stacked autoencoders to mitigate the curse of dimensionality. By injecting less noise into the forward hidden embeddings of more sensitive fine-tuning epochs, EW-FT achieves more targeted local differential privacy perturbations. Extensive experiments on three downstream tasks demonstrate that, while maintaining the same level of privacy protection, our EW-FT achieves higher model accuracy compared with state-of-the-art techniques.