A Unified Framework and a Case Study for Hyperparameter Selection in Machine Learning via Bilevel Optimization

Zhen Li, Yaru Qian, Qingna Li*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Citations (Scopus)

Abstract

Support vector classification (SVC) is a classical and widely used method for classification problems. A regularization parameter, which significantly affects the classification performance, has to be chosen. In this paper, we propose a unified framework for hyperparameter selection in machine learning via bilevel optimization. Specially, we study the case for l2-loss SVC in which the upper-level objective function is the hinge loss function, and the lower-level problems are l2-loss SVC models. Following the same way as in our recent work, we reformulate the bilevel optimization model as a mathematical program with equilibrium constraints (MPEC). Similarly to that in our recent work, the MPEC-tailored version of the Mangasarian-Fromovitz constraint qualification (MPEC-MFCQ) automatically holds at each feasible point of the MPEC. Extensive numerical results verify that the global relaxation cross-validation algorithm for l2-loss SVC (GRCV-l2) enjoys superior generalization performance over almost all the datasets used in this paper.

Original languageEnglish
Title of host publication2022 5th International Conference on Data Science and Information Technology, DSIT 2022 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665498685
DOIs
Publication statusPublished - 2022
Event5th International Conference on Data Science and Information Technology, DSIT 2022 - Shanghai, China
Duration: 22 Jul 202224 Jul 2022

Publication series

Name2022 5th International Conference on Data Science and Information Technology, DSIT 2022 - Proceedings

Conference

Conference5th International Conference on Data Science and Information Technology, DSIT 2022
Country/TerritoryChina
CityShanghai
Period22/07/2224/07/22

Keywords

  • Bilevel optimization
  • C-stationarity
  • Hyperparameter selection
  • Mathematical program with equilibrium constraints
  • Support vector classification

Fingerprint

Dive into the research topics of 'A Unified Framework and a Case Study for Hyperparameter Selection in Machine Learning via Bilevel Optimization'. Together they form a unique fingerprint.

Cite this