AdaSwitch: Adapting switch from Adam to SGDM by exponential function

Optimizers play a critical role in the training of deep neural networks (DNNs). Adam is known for its fast convergence, while Stochastic Gradient Descent with Momentum (SGDM) is valued for its strong generalization capability. However, both exhibit limitations: SGDM often suffers from slow convergence initially, whereas Adam tends to generalize poorly in later stages. To address these issues, we propose AdaSwitch, an optimizer that combines the strengths of both methods, achieving rapid convergence early in training and robust generalization later on. AdaSwitch employs a linear combination based on an exponential function to smoothly transition from Adam to SGDM by adjusting the DNN parameters. We also provide a theoretical convergence guarantee for non-convex settings. The core idea is to express the network parameters θ_t as θ_t=β₃^tθ_t^Adam+(1−β₃^t)θ_t^SGDM, where (β₃∈(0,1)) is the base of the adaptive exponential function. Extensive experiments on various architectures and tasks demonstrate that AdaSwitch outperforms existing methods in image classification, image generation, node classification, and few-shot visual classification, delivering both fast convergence and strong generalization.