Domain-adversarial based model with phonological knowledge for cross-lingual speech recognition

Qingran Zhan, Xiang Xie*, Chenguang Hu, Juan Zuluaga-Gomez, Jing Wang, Haobo Cheng

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)

Abstract

Phonological-based features (articulatory features, AFs) describe the movements of the vocal organ which are shared across languages. This paper investigates a domain-adversarial neural network (DANN) to extract reliable AFs, and different multi-stream techniques are used for crosslingual speech recognition. First, a novel universal phonological attributes definition is proposed for Mandarin, English, German and French. Then a DANN-based AFs detector is trained using source languages (English, German and French). When doing the cross-lingual speech recognition, the AFs detectors are used to transfer the phonological knowledge from source languages (English, German and French) to the target language (Mandarin). Two multi-stream approaches are introduced to fuse the acoustic features and cross-lingual AFs. In addition, the monolingual AFs system (i.e., the AFs are directly extracted from the target language) is also investigated. Experiments show that the performance of the AFs detector can be improved by using convolutional neural networks (CNN) with a domain-adversarial learning method. The multi-head attention (MHA) based multistream can reach the best performance compared to the baseline, cross-lingual adaptation approach, and other approaches. More specifically, the MHA-mode with cross-lingual AFs yields significant improvements over monolingual AFs with the restriction of training data size and, which can be easily extended to other low-resource languages.

Original languageEnglish
Article number3172
JournalElectronics (Switzerland)
Volume10
Issue number24
DOIs
Publication statusPublished - 1 Dec 2021

Keywords

  • Articulatory features
  • Cross-lingual automatic speech recognition (ASR)
  • Domain-adversarial neural network
  • Multi-stream learning

Fingerprint

Dive into the research topics of 'Domain-adversarial based model with phonological knowledge for cross-lingual speech recognition'. Together they form a unique fingerprint.

Cite this