Automatic Assessment Method and Device for Depression Symptom Severity Based on Emotional Facial Expression and Pupil-Wave

Depression is a serious mental disorder, significantly burdens individuals, families, and society. For clinical psychiatrists, assessing the severity of depression is a crucial tool in selecting treatment approaches and evaluating their effectiveness. Although many studies in machine learning have focused on the automatic evaluation of self-rating scales [such as the Beck Depression Inventory-II (BDI-II) and the Patient Health Questionnaire-8 (PHQ-8)], research into the machine learning-based automatic evaluation of the medical clinical assessment scale [such as the Hamilton Depression Scale (HAMD)] has not yet been focused on. In this study, an end-to-end automatic evaluation device for HAMD and Patient Health Questionnaire-9 (PHQ-9) scores was developed. In addition, we constructed a dataset consisting of emotional facial expression videos (eFEVs) signals and emotional pupil-wave (ePW) signals from 65 patients with depression. The dataset has HAMD and PHQ-9 score labels, encompassing two emotional states: sadness and happiness. We built a 3-dimensional convolutional neural network + long short-term memory (3DCNN + LSTM) model framework and a multiscale 1-dimensional convolutional neural network (1DCNN) to learn and extract features from eFEVs and ePW automatically. The results showed that compared with the previous evaluation methods for depression levels, the evaluation precision of HAMD and PHQ-9 has been improved significantly. The results also showed that, in both HAMD and PHQ-9 evaluations, the evaluation precision of eFEVs was superior to ePW, and HAMD is better than PHQ-9. These studies indicated that both emotional facial expressions and ePW can better represent depressive mood in patients with depression, especially emotional facial expressions, and the predictive precision of the medical scale is significantly better than the self-rating scale. This automated assessment method and device can assist doctors in diagnosing depressive symptoms more effectively and serve as an evaluation tool for treatment efficacy.