Decentralized Federated Large Model Tuning over Edge Networks with Zeroth-Order Optimization

Federated tuning of large models is an emerging paradigm that pushes the promising generative AI services into the network edge. However, as model sizes scale up, the conflicts between their intensive resource demands and the naturally limited resource at edge networks significantly constrained the performance of tuning large models over edge networks. In view of these, we propose a fully decentralized federated large model tuning framework with zeroth-order (ZO) optimization, addressing the significant computation and communication costs during the federated learning (FL) process. The proposed framework offers superior performance in dynamic and infrastructure-less edge networks with a theoretical convergence guarantee. Extensive experiments demonstrate the efficacy and outperformance of the proposed framework regarding the communication efficiency and robustness performance.