Correlation between audio-visual enhancement of speech in different noise environments and SNR: A combined behavioral and electrophysiological study

B. Liu*, Y. Lin, X. Gao, J. Dang

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

18 Citations (Scopus)

Abstract

In the present study, we investigated the multisensory gain as the difference of speech recognition accuracies between the audio-visual (AV) and auditory-only (A) conditions, and the multisensory gain as the difference between the event-related potentials (ERPs) evoked under the AV condition and the sum of the ERPs evoked under the A and visual-only (V) conditions in different noise environments. Videos of a female speaker articulating the Chinese monosyllable words accompanied with different levels of pink noise were used as the stimulus materials. The selected signal-to-noise ratios (SNRs) were -16, -12, -8, -4 and 0 dB. Under the A, V and AV conditions the accuracy of the speech recognition was measured and the ERPs evoked under different conditions were analyzed, respectively. The behavioral results showed that the maximum gain as the difference of speech recognition accuracies between the AV and A conditions was at the -12 dB SNR. The ERP results showed that the multisensory gain as the difference between the ERPs evoked under the AV condition and the sum of ERPs evoked under the A and V conditions at the -12 dB SNR was significantly higher than those at the other SNRs in the time window of 130-200. ms in the area from frontal to central region. The multisensory gains in audio-visual speech recognition at different SNRs were not completely accordant with the principle of inverse effectiveness, but confirmed to cross-modal stochastic resonance.

Original languageEnglish
Pages (from-to)145-151
Number of pages7
JournalNeuroscience
Volume247
DOIs
Publication statusPublished - 5 Sept 2013
Externally publishedYes

Keywords

  • Audio-visual speech recognition
  • Cross-modal stochastic resonance
  • ERPs
  • Multisensory gain
  • SNR

Fingerprint

Dive into the research topics of 'Correlation between audio-visual enhancement of speech in different noise environments and SNR: A combined behavioral and electrophysiological study'. Together they form a unique fingerprint.

Cite this