TY - GEN
T1 - A Two-Stage Distillation Method
T2 - 27th International Conference on Asian Language Processing, IALP 2023
AU - Sun, Ruiqing
AU - Jian, Ping
N1 - Publisher Copyright:
© 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - Multi-choice Machine Reading Comprehension (MRC) is a challenging extension of Natural Language Processing (NLP) that requires the ability to comprehend the semantics and logical relationships between entities in a given text. The MRC task has traditionally been viewed as a process of answering questions based on the given text. This single-stage approach has often led the network to concentrate on generating the correct answer, potentially neglecting the comprehension of the text itself. As a result, many prevalent models have faced challenges in performing well on this task when dealing with longer texts. In this paper, we propose a two-stage knowledge distillation method that teaches the model to better comprehend the document by dividing the MRC task into two separate stages. Our experimental results show that the student model, when equipped with our method, achieves significant improvements, demonstrating the effectiveness of our method.
AB - Multi-choice Machine Reading Comprehension (MRC) is a challenging extension of Natural Language Processing (NLP) that requires the ability to comprehend the semantics and logical relationships between entities in a given text. The MRC task has traditionally been viewed as a process of answering questions based on the given text. This single-stage approach has often led the network to concentrate on generating the correct answer, potentially neglecting the comprehension of the text itself. As a result, many prevalent models have faced challenges in performing well on this task when dealing with longer texts. In this paper, we propose a two-stage knowledge distillation method that teaches the model to better comprehend the document by dividing the MRC task into two separate stages. Our experimental results show that the student model, when equipped with our method, achieves significant improvements, demonstrating the effectiveness of our method.
KW - Knowledge Distillation
KW - Multi-choice Machine Reading Comprehension
KW - Semantic Comprehension
KW - Two-stage Distillation
UR - http://www.scopus.com/inward/record.url?scp=85181769631&partnerID=8YFLogxK
U2 - 10.1109/IALP61005.2023.10337099
DO - 10.1109/IALP61005.2023.10337099
M3 - Conference contribution
AN - SCOPUS:85181769631
T3 - Proceedings of 2023 International Conference on Asian Language Processing, IALP 2023
SP - 240
EP - 245
BT - Proceedings of 2023 International Conference on Asian Language Processing, IALP 2023
A2 - Wang, Lei
A2 - Lu, Yanfeng
A2 - Dong, Minghui
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 18 November 2023 through 20 November 2023
ER -