Paper
13 June 2024 STM-Net based spatial-temporal multimodal fusion network for emotion recognition
Lina Li, Wenjie Deng, Shengli Liao, Xue Qiang, Yuying Rong, Ying Yang, Shixuan Liu, Yumei Zhang
Author Affiliations +
Proceedings Volume 13180, International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024); 1318051 (2024) https://doi.org/10.1117/12.3034151
Event: International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024), 2024, Guangzhou, China
Abstract
Emotion recognition plays a vital role in human-computer interaction. However, traditional approaches relying on manual features extraction can get high accuracy results but limited generalization; furthermore results of emotion recognition using a single modality is unreliable. To address these challenges, a multi-modal emotion recognition model called STM-Net, which leverages the spatial and temporal information from electroencephalography (EEG) and eye movement two modalities is proposed based on convolutional neural networks (CNNs) and long short-term memory networks (LSTMs). A CNN-LSTM based model is designed to learn the spatial-temporal features of emotions in EEG signals. For eye-tracking signals, a corresponding CNN-based model is designed for feature extraction. The features from both modalities are fused and fed into a fully connected network for classification, then comprehensive and accurate emotion recognition results are obtained. Experimental results on the SEED-IV multimodal dataset demonstrate the effectiveness of the proposed approach, with accuracy results reaching up to 99.6%, much higher than other similar multimodal emotion recognition models.
(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.
Lina Li, Wenjie Deng, Shengli Liao, Xue Qiang, Yuying Rong, Ying Yang, Shixuan Liu, and Yumei Zhang "STM-Net based spatial-temporal multimodal fusion network for emotion recognition", Proc. SPIE 13180, International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024), 1318051 (13 June 2024); https://doi.org/10.1117/12.3034151
Advertisement
Advertisement
RIGHTS & PERMISSIONS
Get copyright permission  Get copyright permission on Copyright Marketplace
KEYWORDS
Emotion

Electroencephalography

Feature extraction

Eye

Eye models

Feature fusion

Education and training

Back to Top