MERATA: A proof-of-concept framework for integrating multimodal emotion recognition into assistive technology assessment

Source abstract

Assistive technology (AT) assessment is a person-centred and multidisciplinary process that considers users’ functional needs, goals, preferences, psychosocial experiences, and contextual factors when evaluating the suitability and use of an assistive technology device (ATD). Established approaches provide valuable information about users’ perceived psychosocial experiences; however, they do not themselves provide objective multimodal measurements of affective responses during device interaction. This study presents MERATA, a proof-of-concept multimodal emotion recognition framework designed to explore how objective affective information could complement established AT assessment. MERATA integrates facial, vocal, and physiological signals using deep learning-based modality-specific models and feature-level fusion to estimate valence and arousal and classify selected affective states. An exploratory affect-to-indicator layer associates the model-derived emotional outputs with selected PIADS-aligned items. The framework was evaluated using the K-EmoCon multimodal dataset through unimodal, bimodal, and trimodal experiments. Multimodal fusion demonstrated the technical feasibility of integrating complementary affective modalities, with trimodal configurations providing the strongest performance. However, class imbalance and the use of a dataset not collected in an assistive technology context limit interpretation and generalisability. MERATA demonstrates the feasibility of adding objective affective information to established person-centred assistive technology assessment. It is intended to support, rather than replace, clinical reasoning and user-centred decision-making.