Toward Objective Mental Health Screening via Facial Expression Analysis: A Hybrid Deep Learning Proof-of-Concept

Source abstract

Depression remains among the largest contributors to global disease burden, yet early screening in Indonesia still relies heavily on self-report instruments prone to subjective bias. This study presents "Beyond Expression," a hybrid architecture combining a Convolutional Neural Network (ResNet-50), a Swin Transformer, and a Spatiotemporal Attention Module (TSAM) to detect depression-related facial patterns from anonymized 3D facial landmark sequences, intended as a first step toward future objective screening tools rather than a clinically validated instrument. Evaluated via 5-fold stratified cross-validation on 142 participants, the model achieved an average accuracy of 66.95% ± 5.49% and macro F1-score of 0.5102 ± 0.0969. Against a logistic regression baseline (F1 = 0.5091), a paired Wilcoxon signed-rank test and paired t-test found no statistically detectable difference, though the small fold count (n = 5) limits test power. Sensitivity remained low (24.17% ± 20.48%), and a cross-frame Grad-CAM analysis across four representative participants showed Healthy-class activation staying relatively stable across frames (correlation up to 0.80), while Depressed-class activation was markedly less consistent (correlation near zero) and partially collapsed in one case, closely matching the model's low sensitivity. A same-dataset held-out check on withheld DAIC-WOZ test participants (N = 47) gave a single-run sensitivity of 57.14% and specificity of 72.73%, higher than most cross-validation folds but not yet confirmed stable across runs. The authors treat these results as genuine findings from a proof-of-concept stage, distinct from a validated clinical diagnosis, meant to guide refinements in data collection and model design for future iterations ahead of clinical validation.