MS2-GNN: Exploring GNN-Based Multimodal Fusion Network for Depression Detection

Tao Chen, Richang Hong*, Yanrong Guo, Shijie Hao, Bin Hu

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

8 Citations (Scopus)

Abstract

Major depressive disorder (MDD) is one of the most common and severe mental illnesses, posing a huge burden on society and families. Recently, some multimodal methods have been proposed to learn a multimodal embedding for MDD detection and achieved promising performance. However, these methods ignore the heterogeneity/homogeneity among various modalities. Besides, earlier attempts ignore interclass separability and intraclass compactness. Inspired by the above observations, we propose a graph neural network (GNN)-based multimodal fusion strategy named modal-shared modal-specific GNN, which investigates the heterogeneity/homogeneity among various psychophysiological modalities as well as explores the potential relationship between subjects. Specifically, we develop a modal-shared and modal-specific GNN architecture to extract the inter/intramodal characteristics. Furthermore, a reconstruction network is employed to ensure fidelity within the individual modality. Moreover, we impose an attention mechanism on various embeddings to obtain a multimodal compact representation for the subsequent MDD detection task. We conduct extensive experiments on two public depression datasets and the favorable results demonstrate the effectiveness of the proposed algorithm.

Original languageEnglish
Pages (from-to)7749-7759
Number of pages11
JournalIEEE Transactions on Cybernetics
Volume53
Issue number12
DOIs
Publication statusPublished - 1 Dec 2023
Externally publishedYes

Keywords

  • Depression detection
  • graph neural network (GNN)
  • multimodal fusion

Fingerprint

Dive into the research topics of 'MS2-GNN: Exploring GNN-Based Multimodal Fusion Network for Depression Detection'. Together they form a unique fingerprint.

Cite this