Multimodal magnetic resonance image and electroencephalogram constrained fusion algorithm using deep learning

Traditional fusion algorithm is prone to skipping the multimodal attribute of the magnetic resonance image (MRI), lack of consideration of the spatial information of electroencephalogram (EEG) signal, resulting in the ineffective fusion of magnetic resonance image and EEG signal. Thus, constraint fusion algorithm of the multimodal magnetic resonance image and EEG signal using deep learning is proposed. First, the magnetic resonance image and EEG signal are simultaneously acquired for preprocessing. Secondly, the three modal magnetic resonance image features of functional Magnetic Resonance Imaging(fMRI), structural Magnetic Resonance Imaging(sMRI) and Diffusion Tensor Imaging are extracted, respectively, to analyze different modal images by multi-index features fusion. Then, different neural network channels use different coding methods, focusing on the construction of local channel convolutional neural network(CNN) model based on Cross Stage Partial Network transformation, coding network and recognition network design. Finally, the inverse problem of EEG signal is solved by the twomey standardized algorithm, completing the fusion algorithm design. The results show that the CNN model of local channel renders good training effect, the signal preprocessing of the proposed algorithm has obvious effect, the extraction accuracy of magnetic resonance image features of different modals is high, the MI value of image fusion is as high as 0.92, and the image fusion precision is high, and the time consumption is much lower than other algorithms.