Bootstrap Generalization Ability from Loss Landscape Perspective

Huanran Chen; Shitong Shao; Ziyi Wang; Zirui Shang; Jin Chen; Xiaofeng Ji; Xinxiao Wu

doi:10.1007/978-3-031-25075-0_34

Bootstrap Generalization Ability from Loss Landscape Perspective

Huanran Chen, Shitong Shao, Ziyi Wang, Zirui Shang^*, Jin Chen, Xiaofeng Ji, Xinxiao Wu

^*Corresponding author for this work

School of Computer Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Citations (Scopus)

Abstract

Domain generalization aims to learn a model that can generalize well on the unseen test dataset, i.e., out-of-distribution data, which has different distribution from the training dataset. To address domain generalization in computer vision, we introduce the loss landscape theory into this field. Specifically, we bootstrap the generalization ability of the deep learning model from the loss landscape perspective in four aspects, including backbone, regularization, training paradigm, and learning rate. We verify the proposed theory on the NICO++, PACS, and VLCS datasets by doing extensive ablation studies as well as visualizations. In addition, we apply this theory in the ECCV 2022 NICO Challenge1 and achieve the 3rd place without using any domain invariant methods.

Original language	English
Title of host publication	Computer Vision – ECCV 2022 Workshops, Proceedings
Editors	Leonid Karlinsky, Tomer Michaeli, Ko Nishino
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	500-517
Number of pages	18
ISBN (Print)	9783031250743
DOIs	https://doi.org/10.1007/978-3-031-25075-0_34
Publication status	Published - 2023
Event	17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel Duration: 23 Oct 2022 → 27 Oct 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13806 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th European Conference on Computer Vision, ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	23/10/22 → 27/10/22

Keywords

Adaptive learning rate scheduler
Distillation-based fine-tuning paradigm
Domain generalization
Loss landscape
Wide-PyramidNet

Access to Document

10.1007/978-3-031-25075-0_34

Cite this

Chen, H., Shao, S., Wang, Z., Shang, Z., Chen, J., Ji, X., & Wu, X. (2023). Bootstrap Generalization Ability from Loss Landscape Perspective. In L. Karlinsky, T. Michaeli, & K. Nishino (Eds.), Computer Vision – ECCV 2022 Workshops, Proceedings (pp. 500-517). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13806 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-25075-0_34

Chen, Huanran ; Shao, Shitong ; Wang, Ziyi et al. / Bootstrap Generalization Ability from Loss Landscape Perspective. Computer Vision – ECCV 2022 Workshops, Proceedings. editor / Leonid Karlinsky ; Tomer Michaeli ; Ko Nishino. Springer Science and Business Media Deutschland GmbH, 2023. pp. 500-517 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{d74bfd53ddbc488785b2e52f6eb8bc9f,

title = "Bootstrap Generalization Ability from Loss Landscape Perspective",

abstract = "Domain generalization aims to learn a model that can generalize well on the unseen test dataset, i.e., out-of-distribution data, which has different distribution from the training dataset. To address domain generalization in computer vision, we introduce the loss landscape theory into this field. Specifically, we bootstrap the generalization ability of the deep learning model from the loss landscape perspective in four aspects, including backbone, regularization, training paradigm, and learning rate. We verify the proposed theory on the NICO++, PACS, and VLCS datasets by doing extensive ablation studies as well as visualizations. In addition, we apply this theory in the ECCV 2022 NICO Challenge1 and achieve the 3rd place without using any domain invariant methods.",

keywords = "Adaptive learning rate scheduler, Distillation-based fine-tuning paradigm, Domain generalization, Loss landscape, Wide-PyramidNet",

author = "Huanran Chen and Shitong Shao and Ziyi Wang and Zirui Shang and Jin Chen and Xiaofeng Ji and Xinxiao Wu",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th European Conference on Computer Vision, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2023",

doi = "10.1007/978-3-031-25075-0_34",

language = "English",

isbn = "9783031250743",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "500--517",

editor = "Leonid Karlinsky and Tomer Michaeli and Ko Nishino",

booktitle = "Computer Vision – ECCV 2022 Workshops, Proceedings",

address = "Germany",

}

Chen, H, Shao, S, Wang, Z, Shang, Z, Chen, J, Ji, X & Wu, X 2023, Bootstrap Generalization Ability from Loss Landscape Perspective. in L Karlinsky, T Michaeli & K Nishino (eds), Computer Vision – ECCV 2022 Workshops, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13806 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 500-517, 17th European Conference on Computer Vision, ECCV 2022, Tel Aviv, Israel, 23/10/22. https://doi.org/10.1007/978-3-031-25075-0_34

Bootstrap Generalization Ability from Loss Landscape Perspective. / Chen, Huanran; Shao, Shitong; Wang, Ziyi et al.
Computer Vision – ECCV 2022 Workshops, Proceedings. ed. / Leonid Karlinsky; Tomer Michaeli; Ko Nishino. Springer Science and Business Media Deutschland GmbH, 2023. p. 500-517 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13806 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Bootstrap Generalization Ability from Loss Landscape Perspective

AU - Chen, Huanran

AU - Shao, Shitong

AU - Wang, Ziyi

AU - Shang, Zirui

AU - Chen, Jin

AU - Ji, Xiaofeng

AU - Wu, Xinxiao

PY - 2023

Y1 - 2023

N2 - Domain generalization aims to learn a model that can generalize well on the unseen test dataset, i.e., out-of-distribution data, which has different distribution from the training dataset. To address domain generalization in computer vision, we introduce the loss landscape theory into this field. Specifically, we bootstrap the generalization ability of the deep learning model from the loss landscape perspective in four aspects, including backbone, regularization, training paradigm, and learning rate. We verify the proposed theory on the NICO++, PACS, and VLCS datasets by doing extensive ablation studies as well as visualizations. In addition, we apply this theory in the ECCV 2022 NICO Challenge1 and achieve the 3rd place without using any domain invariant methods.

AB - Domain generalization aims to learn a model that can generalize well on the unseen test dataset, i.e., out-of-distribution data, which has different distribution from the training dataset. To address domain generalization in computer vision, we introduce the loss landscape theory into this field. Specifically, we bootstrap the generalization ability of the deep learning model from the loss landscape perspective in four aspects, including backbone, regularization, training paradigm, and learning rate. We verify the proposed theory on the NICO++, PACS, and VLCS datasets by doing extensive ablation studies as well as visualizations. In addition, we apply this theory in the ECCV 2022 NICO Challenge1 and achieve the 3rd place without using any domain invariant methods.

KW - Adaptive learning rate scheduler

KW - Distillation-based fine-tuning paradigm

KW - Domain generalization

KW - Loss landscape

KW - Wide-PyramidNet

UR - http://www.scopus.com/inward/record.url?scp=85150958017&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-25075-0_34

DO - 10.1007/978-3-031-25075-0_34

M3 - Conference contribution

AN - SCOPUS:85150958017

SN - 9783031250743

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 500

EP - 517

BT - Computer Vision – ECCV 2022 Workshops, Proceedings

A2 - Karlinsky, Leonid

A2 - Michaeli, Tomer

A2 - Nishino, Ko

PB - Springer Science and Business Media Deutschland GmbH

T2 - 17th European Conference on Computer Vision, ECCV 2022

Y2 - 23 October 2022 through 27 October 2022

ER -

Chen H, Shao S, Wang Z, Shang Z, Chen J, Ji X et al. Bootstrap Generalization Ability from Loss Landscape Perspective. In Karlinsky L, Michaeli T, Nishino K, editors, Computer Vision – ECCV 2022 Workshops, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 500-517. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-25075-0_34