DePA: Improving Non-autoregressive Machine Translation with Dependency-Aware Decoder

Jiaao Zhan, Qian Chen, Boxing Chen, Wen Wang, Yu Bai, Yang Gao*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Citations (Scopus)

Abstract

Non-autoregressive machine translation (NAT) models have lower translation quality than autoregressive translation (AT) models because NAT decoders do not depend on previous target tokens in the decoder input. We propose a novel and general Dependency-Aware Decoder (DePA) to enhance target dependency modeling in the decoder of fully NAT models from two perspectives: decoder self-attention and decoder input. First, we propose an autoregressive forward-backward pre-training phase before NAT training, which enables the NAT decoder to gradually learn bidirectional target dependencies for the final NAT training. Second, we transform the decoder input from the source language representation space to the target language representation space through a novel attentive transformation process, which enables the decoder to better capture target dependencies. DePA can be applied to any fully NAT models. Extensive experiments show that DePA consistently improves highly competitive and state-of-the-art fully NAT models on widely used WMT and IWSLT benchmarks by up to 1.88 BLEU gain, while maintaining the inference latency comparable to other fully NAT models.1

Original languageEnglish
Title of host publication20th International Conference on Spoken Language Translation, IWSLT 2023 - Proceedings of the Conference
EditorsElizabeth Salesky, Marcello Federico, Marine Carpuat
PublisherAssociation for Computational Linguistics
Pages478-490
Number of pages13
ISBN (Electronic)9781959429845
Publication statusPublished - 2023
Event20th International Conference on Spoken Language Translation, IWSLT 2023 - Hybrid, Toronto, Canada
Duration: 13 Jul 202314 Jul 2023

Publication series

Name20th International Conference on Spoken Language Translation, IWSLT 2023 - Proceedings of the Conference

Conference

Conference20th International Conference on Spoken Language Translation, IWSLT 2023
Country/TerritoryCanada
CityHybrid, Toronto
Period13/07/2314/07/23

Fingerprint

Dive into the research topics of 'DePA: Improving Non-autoregressive Machine Translation with Dependency-Aware Decoder'. Together they form a unique fingerprint.

Cite this