TY - JOUR
T1 - A New Framework of Collaborative Learning for Adaptive Metric Distillation
AU - Liu, Hao
AU - Ye, Mang
AU - Wang, Yan
AU - Zhao, Sanyuan
AU - Li, Ping
AU - Shen, Jianbing
N1 - Publisher Copyright:
© 2012 IEEE.
PY - 2024/6/1
Y1 - 2024/6/1
N2 - This article presents a new adaptive metric distillation approach that can significantly improve the student networks' backbone features, along with better classification results. Previous knowledge distillation (KD) methods usually focus on transferring the knowledge across the classifier logits or feature structure, ignoring the excessive sample relations in the feature space. We demonstrated that such a design greatly limits performance, especially for the retrieval task. The proposed collaborative adaptive metric distillation (CAMD) has three main advantages: 1) the optimization focuses on optimizing the relationship between key pairs by introducing the hard mining strategy into the distillation framework; 2) it provides an adaptive metric distillation that can explicitly optimize the student feature embeddings by applying the relation in the teacher embeddings as supervision; and 3) it employs a collaborative scheme for effective knowledge aggregation. Extensive experiments demonstrated that our approach sets a new state-of-the-art in both the classification and retrieval tasks, outperforming other cutting-edge distillers under various settings.
AB - This article presents a new adaptive metric distillation approach that can significantly improve the student networks' backbone features, along with better classification results. Previous knowledge distillation (KD) methods usually focus on transferring the knowledge across the classifier logits or feature structure, ignoring the excessive sample relations in the feature space. We demonstrated that such a design greatly limits performance, especially for the retrieval task. The proposed collaborative adaptive metric distillation (CAMD) has three main advantages: 1) the optimization focuses on optimizing the relationship between key pairs by introducing the hard mining strategy into the distillation framework; 2) it provides an adaptive metric distillation that can explicitly optimize the student feature embeddings by applying the relation in the teacher embeddings as supervision; and 3) it employs a collaborative scheme for effective knowledge aggregation. Extensive experiments demonstrated that our approach sets a new state-of-the-art in both the classification and retrieval tasks, outperforming other cutting-edge distillers under various settings.
KW - Collaborative learning
KW - deep neural networks
KW - knowledge distillation (KD)
KW - model compression
UR - http://www.scopus.com/inward/record.url?scp=85149361906&partnerID=8YFLogxK
U2 - 10.1109/TNNLS.2022.3226569
DO - 10.1109/TNNLS.2022.3226569
M3 - Article
AN - SCOPUS:85149361906
SN - 2162-237X
VL - 35
SP - 8266
EP - 8277
JO - IEEE Transactions on Neural Networks and Learning Systems
JF - IEEE Transactions on Neural Networks and Learning Systems
IS - 6
ER -