Abstract
The Non-Autoregressive Transformer, due to its low inference latency, has attracted much attention from researchers. Although, the performance of the non-autoregressive transformer has been significantly improved in recent years, there is still a gap between the non-autoregressive transformer and the autoregressive transformer. Considering the success of localness on the autoregressive transformer, in this work, we consider incorporating localness into the non-autoregressive transformer. Specifically, we design a dynamic mask matrix according to the query tokens, key tokens, and relative distance, and unify the localness module for self-attention and cross-attention module. We conduct experiments on several benchmark tasks, and the results show that our model can significantly improve the performance of the non-autoregressive transformer.
Original language | English |
---|---|
Article number | 125 |
Journal | ACM Transactions on Asian and Low-Resource Language Information Processing |
Volume | 22 |
Issue number | 5 |
DOIs | |
Publication status | Published - 8 May 2023 |
Keywords
- Non-autoregressive
- attention module
- localness
- translation