Encoding and Decoding of Chinese Phonemes Based on MEG Signals

Phonemes, the smallest phonetic units of speech sound, bridge the connection between neural recordings and words. Decoding phonemes from non-invasive Electro-/Magnetoencephalography (EEG/MEG) has been demonstrated to be feasible. However, previous studies mainly focused on decoding with EEG/MEG that was evoked by isolated phonemes or syllable stimuli, which do not align with the continuous speech scenarios encountered in practical applications. In this study, we investigated the neural encoding and decoding of Mandarin Chinese phonemes in continuous speech using MEG signals. Firstly, a Vector Quantized Variational Autoencoder (VQ-VAE) speech model was trained to extract phoneme features from continuous speech. Subsequently, representational similarity analysis (RSA) was performed on these extracted features along with their corresponding MEG data to explore the temporal patterns of phoneme representation in MEG. Finally, neural networks were utilized to reconstruct phoneme features from MEG signals, and the decoding performance at both the segment and frame levels was evaluated. The RSA results show that Chinese phonemes exhibit significant representation in MEG signals from 160 ms to 300 ms after the phoneme onset. The decoding results show that at the segment level (3 seconds), using phoneme features decoded from MEG signals for retrieval tasks (candidate number: 987), a top-10 accuracy of 41.57% can be achieved. At the frame level, the six-class accuracy for classifying phoneme articulation manners is 36.16%.