A Unified Framework and a Case Study for Hyperparameter Selection in Machine Learning via Bilevel Optimization

Support vector classification (SVC) is a classical and widely used method for classification problems. A regularization parameter, which significantly affects the classification performance, has to be chosen. In this paper, we propose a unified framework for hyperparameter selection in machine learning via bilevel optimization. Specially, we study the case for l2-loss SVC in which the upper-level objective function is the hinge loss function, and the lower-level problems are l2-loss SVC models. Following the same way as in our recent work, we reformulate the bilevel optimization model as a mathematical program with equilibrium constraints (MPEC). Similarly to that in our recent work, the MPEC-tailored version of the Mangasarian-Fromovitz constraint qualification (MPEC-MFCQ) automatically holds at each feasible point of the MPEC. Extensive numerical results verify that the global relaxation cross-validation algorithm for l2-loss SVC (GRCV-l2) enjoys superior generalization performance over almost all the datasets used in this paper.