EESCAN: An EEG and Eye Movement Fusion Network Combining Intra-modal Self-attention and Inter-modal Bidirectional Interaction for Dual RSVP Classification Task

Objective: Rapid Serial Visual Presentation (RSVP) had been applied to human-computer interactions such as spelling, device control and so on. Traditional single RSVP paradigm detects targets in only one image stream, which may lead to missed or false detection. Dual RSVP paradigm can enhance the classification robustness through specific encoding to increase the number of targets. Methods: To enhance the classification performance of dual RSVP, an EEG-EM Self Attention and Cross Attention Network (EESCAN) is proposed. EEG and EM signals undergo a symmetric two-stream backbone. Each stream consists a convolution module and a self-attention module to extract local and global features. Subsequently, an inter-modal bidirectional interaction module is proposed to provide complementary information between EEG and EM modality. Finally, dynamic reweighting and fusion module is employed to dynamically adjust the sample-level weights according to the contribution of EEG and EM features. Moreover, a VR-based dual RSVP virtual robotic arm control system using the proposed algorithm is then designed to achieve gaze-independent device control. Results: EEG and EM data from 21 subjects were collected and analyzed. The proposed network and system achieved better performance than existing decoding methods and single-modal baselines. Ablation experiments and visualization results further verified the effectiveness of each proposed module. Conclulsion: EESCAN network is proposed for the dual RSVP paradigm that integrates intra-modal self-attention, inter-modal bidirectional interaction, and dynamic fusion to achieve EEG-EM fusion. Significance: EESCAN markedly improves classification performance of dual RSVP-based BCIs. This gaze-independent control system is suitable for patients with restricted gaze shifts.