RS-BERT: Pre-training radical enhanced sense embedding for Chinese word sense disambiguation

Xiaofeng Zhou, Heyan Huang*, Zewen Chi, Mucheng Ren, Yang Gao

*此作品的通讯作者

科研成果: 期刊稿件文章同行评审

摘要

Word sense disambiguation is a crucial task to test whether a model can perform deep understanding. Nowadays, the rise of pre-trained language models facilitates substantial success in such tasks. However, most current pre-training tasks are taken in place of the token level while ignoring the linguistic sense of the tokens themselves. Thus it is questionable whether the token-predicting objectives are enough to learn the polysemy and disambiguate senses. To explore this question, we introduce RS-BERT, a radical enhanced sense embedding model with a novel pre-training objective, sense-aware language modeling, which introduces additional sense-level information to the model. For each training step, we first predict the senses and then update the model given the predicted senses. During training, we alternately perform the above two steps in an expectation–maximization manner. Besides, we also introduce radical information to RS-BERT at the beginning of pre-training. We conduct experiments on two Chinese word sense disambiguation datasets. Experimental results show that RS-BERT is competitive. When combined with other dedicated adaptions for specific datasets, RS-BERT shows impressive performance. Moreover, our analysis shows that RS-BERT successfully clusters Chinese characters into various senses. The experiment results demonstrate that the token-predicting objectives are not enough and the sense-level objective performs better for polysemy and sense disambiguation.

源语言英语
文章编号103740
期刊Information Processing and Management
61
4
DOI
出版状态已出版 - 7月 2024

指纹

探究 'RS-BERT: Pre-training radical enhanced sense embedding for Chinese word sense disambiguation' 的科研主题。它们共同构成独一无二的指纹。

引用此

Zhou, X., Huang, H., Chi, Z., Ren, M., & Gao, Y. (2024). RS-BERT: Pre-training radical enhanced sense embedding for Chinese word sense disambiguation. Information Processing and Management, 61(4), 文章 103740. https://doi.org/10.1016/j.ipm.2024.103740