TY - GEN
T1 - XLM-E
T2 - 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022
AU - Chi, Zewen
AU - Huang, Shaohan
AU - Dong, Li
AU - Ma, Shuming
AU - Zheng, Bo
AU - Singhal, Saksham
AU - Bajaj, Payal
AU - Song, Xia
AU - Mao, Xian Ling
AU - Huang, Heyan
AU - Wei, Furu
N1 - Publisher Copyright:
© 2022 Association for Computational Linguistics.
PY - 2022
Y1 - 2022
N2 - In this paper, we introduce ELECTRA-style tasks (Clark et al., 2020b) to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.
AB - In this paper, we introduce ELECTRA-style tasks (Clark et al., 2020b) to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.
UR - http://www.scopus.com/inward/record.url?scp=85140387868&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85140387868
T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics
SP - 6170
EP - 6182
BT - ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)
A2 - Muresan, Smaranda
A2 - Nakov, Preslav
A2 - Villavicencio, Aline
PB - Association for Computational Linguistics (ACL)
Y2 - 22 May 2022 through 27 May 2022
ER -