摘要
In this paper, we introduce ELECTRA-style tasks (Clark et al., 2020b) to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.
| 源语言 | 英语 |
|---|---|
| 主期刊名 | ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers) |
| 编辑 | Smaranda Muresan, Preslav Nakov, Aline Villavicencio |
| 出版商 | Association for Computational Linguistics (ACL) |
| 页 | 6170-6182 |
| 页数 | 13 |
| ISBN（电子版） | 9781955917216 |
| 出版状态 | 已出版 - 2022 |
| 活动 | 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - Dublin, 爱尔兰 期限: 22 5月 2022 → 27 5月 2022 |
出版系列
| 姓名 | Proceedings of the Annual Meeting of the Association for Computational Linguistics |
|---|---|
| 卷 | 1 |
| ISSN（印刷版） | 0736-587X |
会议
| 会议 | 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 |
|---|---|
| 国家/地区 | 爱尔兰 |
| 市 | Dublin |
| 时期 | 22/05/22 → 27/05/22 |
指纹
探究 'XLM-E: Cross-lingual Language Model Pre-training via ELECTRA' 的科研主题。它们共同构成独一无二的指纹。引用此
Chi, Z., Huang, S., Dong, L., Ma, S., Zheng, B., Singhal, S., Bajaj, P., Song, X., Mao, X. L., Huang, H., & Wei, F. (2022). XLM-E: Cross-lingual Language Model Pre-training via ELECTRA. 在 S. Muresan, P. Nakov, & A. Villavicencio (编辑), ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers) (页码 6170-6182). (Proceedings of the Annual Meeting of the Association for Computational Linguistics; 卷 1). Association for Computational Linguistics (ACL).