Enhancing Zero-Shot Translation in Multilingual Neural Machine Translation: Focusing on Obtaining Location-Agnostic Representations

Jiarui Zhang; Heyan Huang; Yue Hu; Ping Guo

doi:10.1007/978-3-031-72350-6_13

Enhancing Zero-Shot Translation in Multilingual Neural Machine Translation: Focusing on Obtaining Location-Agnostic Representations

Jiarui Zhang^*, Heyan Huang^*, Yue Hu, Ping Guo

^*Corresponding author for this work

School of Computer Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In the field of multilingual neural machine translation, a notable challenge is zero-shot translation, where a model translates languages that it has not been trained on. This often results in poor translation quality, mainly because the model’s internal language representations are too specific to its training languages. We illustrate that the positional relationship to input tokens is a primary factor contributing to the language-specific representations. We find a solution by modifying the model’s structure, specifically by removing certain connections in its encoder layer. This simple change significantly improves the quality of zero-shot translations, with an increase of up to 11.1 BLEU points, a measure of translation accuracy. Importantly, this improvement does not affect the quality of translations for the languages the model was trained on. Besides, our method facilitates the seamless incorporation of new languages, significantly broadening the scope of translation coverage.

Original language	English
Title of host publication	Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings
Editors	Michael Wand, Jürgen Schmidhuber, Michael Wand, Kristína Malinovská, Jürgen Schmidhuber, Igor V. Tetko, Igor V. Tetko
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	194-208
Number of pages	15
ISBN (Print)	9783031723490
DOIs	https://doi.org/10.1007/978-3-031-72350-6_13
Publication status	Published - 2024
Event	33rd International Conference on Artificial Neural Networks, ICANN 2024 - Lugano, Switzerland Duration: 17 Sept 2024 → 20 Sept 2024

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	15022 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	33rd International Conference on Artificial Neural Networks, ICANN 2024
Country/Territory	Switzerland
City	Lugano
Period	17/09/24 → 20/09/24

Keywords

location-agnostic representations
multilingual neural machine translation
removing certain connections
zero-shot translation

Access to Document

10.1007/978-3-031-72350-6_13

Cite this

Zhang, J., Huang, H., Hu, Y., & Guo, P. (2024). Enhancing Zero-Shot Translation in Multilingual Neural Machine Translation: Focusing on Obtaining Location-Agnostic Representations. In M. Wand, J. Schmidhuber, M. Wand, K. Malinovská, J. Schmidhuber, I. V. Tetko, & I. V. Tetko (Eds.), Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings (pp. 194-208). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15022 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-72350-6_13

Zhang, Jiarui ; Huang, Heyan ; Hu, Yue et al. / Enhancing Zero-Shot Translation in Multilingual Neural Machine Translation : Focusing on Obtaining Location-Agnostic Representations. Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings. editor / Michael Wand ; Jürgen Schmidhuber ; Michael Wand ; Kristína Malinovská ; Jürgen Schmidhuber ; Igor V. Tetko ; Igor V. Tetko. Springer Science and Business Media Deutschland GmbH, 2024. pp. 194-208 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{33b724ad2b5142b19e67275f5bf8eba3,

title = "Enhancing Zero-Shot Translation in Multilingual Neural Machine Translation: Focusing on Obtaining Location-Agnostic Representations",

abstract = "In the field of multilingual neural machine translation, a notable challenge is zero-shot translation, where a model translates languages that it has not been trained on. This often results in poor translation quality, mainly because the model{\textquoteright}s internal language representations are too specific to its training languages. We illustrate that the positional relationship to input tokens is a primary factor contributing to the language-specific representations. We find a solution by modifying the model{\textquoteright}s structure, specifically by removing certain connections in its encoder layer. This simple change significantly improves the quality of zero-shot translations, with an increase of up to 11.1 BLEU points, a measure of translation accuracy. Importantly, this improvement does not affect the quality of translations for the languages the model was trained on. Besides, our method facilitates the seamless incorporation of new languages, significantly broadening the scope of translation coverage.",

keywords = "location-agnostic representations, multilingual neural machine translation, removing certain connections, zero-shot translation",

author = "Jiarui Zhang and Heyan Huang and Yue Hu and Ping Guo",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 33rd International Conference on Artificial Neural Networks, ICANN 2024 ; Conference date: 17-09-2024 Through 20-09-2024",

year = "2024",

doi = "10.1007/978-3-031-72350-6_13",

language = "English",

isbn = "9783031723490",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "194--208",

editor = "Michael Wand and J{\"u}rgen Schmidhuber and Michael Wand and Krist{\'i}na Malinovsk{\'a} and J{\"u}rgen Schmidhuber and Tetko, {Igor V.} and Tetko, {Igor V.}",

booktitle = "Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings",

address = "Germany",

}

Zhang, J, Huang, H, Hu, Y & Guo, P 2024, Enhancing Zero-Shot Translation in Multilingual Neural Machine Translation: Focusing on Obtaining Location-Agnostic Representations. in M Wand, J Schmidhuber, M Wand, K Malinovská, J Schmidhuber, IV Tetko & IV Tetko (eds), Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15022 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 194-208, 33rd International Conference on Artificial Neural Networks, ICANN 2024, Lugano, Switzerland, 17/09/24. https://doi.org/10.1007/978-3-031-72350-6_13

Enhancing Zero-Shot Translation in Multilingual Neural Machine Translation: Focusing on Obtaining Location-Agnostic Representations. / Zhang, Jiarui; Huang, Heyan; Hu, Yue et al.
Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings. ed. / Michael Wand; Jürgen Schmidhuber; Michael Wand; Kristína Malinovská; Jürgen Schmidhuber; Igor V. Tetko; Igor V. Tetko. Springer Science and Business Media Deutschland GmbH, 2024. p. 194-208 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15022 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Enhancing Zero-Shot Translation in Multilingual Neural Machine Translation

T2 - 33rd International Conference on Artificial Neural Networks, ICANN 2024

AU - Zhang, Jiarui

AU - Huang, Heyan

AU - Hu, Yue

AU - Guo, Ping

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

PY - 2024

Y1 - 2024

N2 - In the field of multilingual neural machine translation, a notable challenge is zero-shot translation, where a model translates languages that it has not been trained on. This often results in poor translation quality, mainly because the model’s internal language representations are too specific to its training languages. We illustrate that the positional relationship to input tokens is a primary factor contributing to the language-specific representations. We find a solution by modifying the model’s structure, specifically by removing certain connections in its encoder layer. This simple change significantly improves the quality of zero-shot translations, with an increase of up to 11.1 BLEU points, a measure of translation accuracy. Importantly, this improvement does not affect the quality of translations for the languages the model was trained on. Besides, our method facilitates the seamless incorporation of new languages, significantly broadening the scope of translation coverage.

AB - In the field of multilingual neural machine translation, a notable challenge is zero-shot translation, where a model translates languages that it has not been trained on. This often results in poor translation quality, mainly because the model’s internal language representations are too specific to its training languages. We illustrate that the positional relationship to input tokens is a primary factor contributing to the language-specific representations. We find a solution by modifying the model’s structure, specifically by removing certain connections in its encoder layer. This simple change significantly improves the quality of zero-shot translations, with an increase of up to 11.1 BLEU points, a measure of translation accuracy. Importantly, this improvement does not affect the quality of translations for the languages the model was trained on. Besides, our method facilitates the seamless incorporation of new languages, significantly broadening the scope of translation coverage.

KW - location-agnostic representations

KW - multilingual neural machine translation

KW - removing certain connections

KW - zero-shot translation

UR - http://www.scopus.com/inward/record.url?scp=85205310723&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-72350-6_13

DO - 10.1007/978-3-031-72350-6_13

M3 - Conference contribution

AN - SCOPUS:85205310723

SN - 9783031723490

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 194

EP - 208

BT - Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings

A2 - Wand, Michael

A2 - Schmidhuber, Jürgen

A2 - Wand, Michael

A2 - Malinovská, Kristína

A2 - Schmidhuber, Jürgen

A2 - Tetko, Igor V.

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 17 September 2024 through 20 September 2024

ER -

Zhang J, Huang H, Hu Y, Guo P. Enhancing Zero-Shot Translation in Multilingual Neural Machine Translation: Focusing on Obtaining Location-Agnostic Representations. In Wand M, Schmidhuber J, Wand M, Malinovská K, Schmidhuber J, Tetko IV, Tetko IV, editors, Artificial Neural Networks and Machine Learning – ICANN 2024 - 33rd International Conference on Artificial Neural Networks, Proceedings. Springer Science and Business Media Deutschland GmbH. 2024. p. 194-208. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-72350-6_13