Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) for Depression Recognition with Facial Expression and Pupil Diameter

Facial expressions have been widely used for depression recognition because it is intuitive and convenient to access. Pupil diameter contains rich emotional information that is already reflected in facial video streams. However, the spatiotemporal correlation between pupillary changes and facial behavior changes induced by emotional stimuli has not been explored in existing studies. This paper presents a novel multimodal fusion algorithm - Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) to optimize the feature space and build a more robust depression recognition model, which innovatively combines the spatiotemporal relevance and complementarity between facial expression and pupil diameter features. TSTCCA explores the interaction between trials and obtains an effective fusion representation of two modalities from a trial subset related to depression. The experimental results show that TSTCCA achieves the highest accuracy of 78.81% with the subset of 25 trials.