跳到主要导航 跳到搜索 跳到主要内容

Depressive semantic awareness from vlog facial and vocal streams via spatio-temporal transformer

  • Yongfeng Tao
  • , Minqiang Yang*
  • , Yushan Wu
  • , Kevin Lee
  • , Adrienne Kline
  • , Bin Hu*
  • *此作品的通讯作者
  • Lanzhou University
  • University of New South Wales
  • Northwestern University

科研成果: 期刊稿件 › 文章 › 同行评审

摘要

With the rapid growth of information transmission via the Internet, efforts have been made to reduce network load to promote efficiency. One such application is semantic computing, which can extract and process semantic communication. Social media has enabled users to share their current emotions, opinions, and life events through their mobile devices. Notably, people suffering from mental health problems are more willing to share their feelings on social networks. Therefore, it is necessary to extract semantic information from social media (vlog data) to identify abnormal emotional states to facilitate early identification and intervention. Most studies do not consider spatio-temporal information when fusing multimodal information to identify abnormal emotional states such as depression. To solve this problem, this paper proposes a spatio-temporal squeeze transformer method for the extraction of semantic features of depression. First, a module with spatio-temporal data is embedded into the transformer encoder, which is utilized to obtain a representation of spatio-temporal features. Second, a classifier with a voting mechanism is designed to encourage the model to classify depression and non-depression effectively. Experiments are conducted on the D-Vlog dataset. The results show that the method is effective, and the accuracy rate can reach 70.70%. This work provides scaffolding for future work in the detection of affect recognition in semantic communication based on social media vlog data.

源语言英语
页(从-至)577-585
页数9
期刊Digital Communications and Networks
卷10
期3
DOI
出版状态已出版 - 6月 2024
已对外发布是

联合国可持续发展目标

此成果有助于实现下列可持续发展目标:

  1. 可持续发展目标 3 - 良好健康与福祉
    可持续发展目标 3 良好健康与福祉

学术指纹

探究 'Depressive semantic awareness from vlog facial and vocal streams via spatio-temporal transformer' 的科研主题。它们共同构成独一无二的学术指纹。

引用此