融合音画同步的唇形合成研究

Cong Jin, Jie Wang, Zichun Guo*, Jing Wang

*此作品的通讯作者

科研成果: 期刊稿件文章同行评审

摘要

With the flourishing development of video-based information dissemination, audio and video synchronization is gradually becoming an important standard for measuring video quality. Deep synthesis technology has been entering the public's view in the international communication field, and lip-sync technology integrating audio and video synchronization has attracted more and more attention. The existing lip-synthesis models are mainly based on lip-synthesis of static images, which are not effective for synthesis of dynamic videos, and most of them use English datasets for training which results in poor synthesis of Chinese Mandarin. To address these problems, this paper conducted optimization experiments on the Wav2Lip lip synthesis model in Chinese context based on its research foundation, and tested the effect of different routes of training models through multiple sets of experiments, which provided important reference values for the subsequent Wav2Lip series research. This study realized lip synthesis from speech-driven to text-driven, discussed the application of lip synthesis in multiple fields such as virtual digital human, and laid the foundation for the broader application and development of lip synthesis technology.

投稿的翻译标题Lipsynthesis incorporating audio-visual synchronisation
源语言繁体中文
页(从-至)397-405
页数9
期刊Chinese Journal of Intelligent Science and Technology
5
3
DOI
出版状态已出版 - 15 9月 2023

关键词

  • artificial intelligence
  • computer visualization
  • deep learning
  • lip generation
  • synchronization of audio and video

指纹

探究 '融合音画同步的唇形合成研究' 的科研主题。它们共同构成独一无二的指纹。

引用此