Subject-Adaptive Multimodal Fusion for Robust Affect Detection: Bridging the Gap Between Laboratory Baselines and Real-World Physiological Noise Using CNN-LSTM With Attention Mechanism

Source abstract

Wearable physiological signal analysis offers transformative potential for mental health monitoring, yet existing affect detection systems demonstrate limited generalization across individuals due to inter-subject variability and environmental noise. This research addresses the critical gap between controlled laboratory performance and real-world applicability by developing a subject-adaptive multimodal fusion framework. We propose a novel hybrid CNN-LSTM architecture with integrated cross-modal attention mechanisms that dynamically weights contributions from six physiological modalities: electrocardiogram, electrodermal activity, respiratory, electromyogram, temperature, and accelerometer signals. Employing rigorous Leave-One-Subject-Out validation on the WESAD dataset (n=15), our framework achieves 96.2% global accuracy and 98.6% AUC-ROC in binary stress detection, representing a 3-7% improvement over existing benchmarks. Notably, eight subjects achieved perfect classification (AUC=1.000), while all others exceeded 90.5% accuracy. The attention mechanism provides interpretable feature weighting, identifying cardiac (42%) and respiratory (28%) signals as primary stress indicators while adapting to individual response patterns. These findings establish a new state-of-the-art for multimodal affect detection with enhanced generalization capabilities. The framework's practical significance lies in its potential for developing personalized mental health monitoring systems that maintain accuracy across diverse populations while offering explainable insights into physiological stress responses, thereby advancing toward clinically deployable wearable technologies.