CRAT-XML: Contrastive Representation Adversarial Training for Extremely Multi-Label Text Classification

Extreme multi-label text classification is a very important and challenging problem in this age of widespread internet access, with a wide range of application scenarios, such as web tagging, legal document annotation, commodity classification, etc. Most of the existing state-of-the-art methods are based on deep learning, however, there are still two problems: 1) the vector representation generated by most pre-trained models suffers from anisotropy and uneven distribution, which has a significant impact on the XMC task. 2) existing models are large in size and use many models for integration. Some of them even took hundreds of hours to train. This seriously affects the efficiency of the experiments. Therefore, in this paper, we propose CRAT-XML, which uses contrast adversarial learning to optimize text representation and enhance the acquisition of dependency relations between text and labels, thus reducing the need for integration at the representation level and achieving relatively high accuracy under low-resource and low-time conditions. Experimental results demonstrate that our model achieves SOTA results on a single model, while achieving a large reduction in training time and model size.