CausalCEM: Self-improved causal counterfactual emotion modeling for multimodal physiological signals

Multimodal physiological signals originating from the central and autonomic nervous systems (CNS/ANS) hold unique advantages for emotion recognition. However, existing methods struggle to effectively model the inter-system and temporal synergy inherent in these heterogeneous signals. Furthermore, existing neural models are typically opaque, hindering the interpretation of underlying causal dependencies and the prediction of intervention outcomes. To address these challenges, we propose Causal Counterfactual Emotion Modeling (CausalCEM), a novel self-improved framework for multimodal physiological signal-based emotion recognition. CausalCEM achieves the research goal of revealing underlying neural system synergy mechanisms and enhancing model interpretability and robustness by unifying causal discovery and counterfactual reasoning to drive the physiological emotion recognition process. Specifically, we formulate the physiological generation process as a Structural Causal Model (SCM) to uncover the directed causal topology and quantitative influence among CNS and ANS variables. Guided by this structure, we introduce a Causal-Informed Temporal Gated Network (CITGN) that fuses causal priors with temporal dynamics for robust prediction. Furthermore, we devise a Physiological-constraint Counterfactual (PhysCF) phase to generate plausible counterfactual samples via topology-guided interventions, driving a self-improved training strategy to iteratively refine the framework. This continuous regularization forces the model to adhere to underlying physiological causality, effectively mitigating reliance on spurious correlations. Experimental results demonstrate that CausalCEM significantly improves emotion recognition performance by effectively modeling the underlying causal synergy and facilitating counterfactual prediction.