Penalty scheme-based Generalized LevenbergMarquardt Method in Hyperparameter Selection in Support Vector Regression

Support vector regression (SVR) serves as a robust regression mechanism within the realm of machine learning, wherein its efficacy is significantly influenced by the choice of hyperparameters. In this paper, we introduce a bilevel optimization framework dedicated to the selection of hyperparameters in SVR, subsequently reformulating the model into a mathematical program with equilibrium constraints (MPEC). Our first contribution involves presenting a novel penalty scheme-based generalized Levenberg-Marquardt method (P-GLM) for solving this problem and providing corresponding convergence results. Distinct from traditional approaches where the subproblem in penalty scheme is treated as a blackbox, we propose P- GLM algorithm to solve the subproblem. Numerical experiments substantiate the superiority of P-GLM over penalty method executed via the fmincon function in MATLAB.