Self-Tuned Regularized Federated Methods With Guarantees for Optimal Solution Selection

We study a hierarchical federated learning (FL) problem arising in over-parameterized learning and ill-posed optimization, where clients seek a solution that minimizes a secondary loss function among multiple optimal solutions of a primary distributed learning problem. First, when the inner-level objective is convex and the outer-level objective is convex or strongly convex, we propose a self-tuned regularized federated averaging method (StR-FedAvg). Second, for nonconvex outer-level objectives, we develop a two-loop FL scheme employing an inexact projected first-order method and StR-FedAvg with an iteratively updated regularization parameter. We establish communication complexity guarantees for both settings and preliminary experiments validate our theoretical findings.