A New Framework of Collaborative Learning for Adaptive Metric Distillation

Hao Liu, Mang Ye, Yan Wang, Sanyuan Zhao*, Ping Li, Jianbing Shen*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)

Abstract

This article presents a new adaptive metric distillation approach that can significantly improve the student networks' backbone features, along with better classification results. Previous knowledge distillation (KD) methods usually focus on transferring the knowledge across the classifier logits or feature structure, ignoring the excessive sample relations in the feature space. We demonstrated that such a design greatly limits performance, especially for the retrieval task. The proposed collaborative adaptive metric distillation (CAMD) has three main advantages: 1) the optimization focuses on optimizing the relationship between key pairs by introducing the hard mining strategy into the distillation framework; 2) it provides an adaptive metric distillation that can explicitly optimize the student feature embeddings by applying the relation in the teacher embeddings as supervision; and 3) it employs a collaborative scheme for effective knowledge aggregation. Extensive experiments demonstrated that our approach sets a new state-of-the-art in both the classification and retrieval tasks, outperforming other cutting-edge distillers under various settings.

Original languageEnglish
Pages (from-to)8266-8277
Number of pages12
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume35
Issue number6
DOIs
Publication statusPublished - 1 Jun 2024

Keywords

  • Collaborative learning
  • deep neural networks
  • knowledge distillation (KD)
  • model compression

Fingerprint

Dive into the research topics of 'A New Framework of Collaborative Learning for Adaptive Metric Distillation'. Together they form a unique fingerprint.

Cite this