Revisit the Power of Vanilla Knowledge Distillation: from Small Scale to Large Scale

Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu*, Chang Xu, Yunhe Wang*

*此作品的通讯作者

科研成果: 期刊稿件会议文章同行评审

2 引用 (Scopus)

摘要

The tremendous success of large models trained on extensive datasets demonstrates that scale is a key ingredient in achieving superior results. Therefore, the reflection on the rationality of designing knowledge distillation (KD) approaches for limited-capacity architectures solely based on small-scale datasets is now deemed imperative. In this paper, we identify the small data pitfall that presents in previous KD methods, which results in the underestimation of the power of vanilla KD framework on large-scale datasets such as ImageNet-1K. Specifically, we show that employing stronger data augmentation techniques and using larger datasets can directly decrease the gap between vanilla KD and other meticulously designed KD variants. This highlights the necessity of designing and evaluating KD approaches in the context of practical scenarios, casting off the limitations of small-scale datasets. Our investigation of the vanilla KD and its variants in more complex schemes, including stronger training strategies and different model capacities, demonstrates that vanilla KD is elegantly simple but astonishingly effective in large-scale scenarios. Without bells and whistles, we obtain state-of-the-art ResNet-50, ViT-S, and ConvNeXtV2-T models for ImageNet, which achieve 83.1%, 84.3%, and 85.0% top-1 accuracy, respectively. PyTorch code and checkpoints can be found at https://github.com/Hao840/vanillaKD.

源语言英语
期刊Advances in Neural Information Processing Systems
36
出版状态已出版 - 2023
活动37th Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, 美国
期限: 10 12月 202316 12月 2023

指纹

探究 'Revisit the Power of Vanilla Knowledge Distillation: from Small Scale to Large Scale' 的科研主题。它们共同构成独一无二的指纹。

引用此