| 作者 | 单位 | | NIU Jian-wei | School of Mechanical Engineering, University of Science and Technology Beijing, Beijing 100083, China | | AN Yue-qi | School of Mechanical Engineering, University of Science and Technology Beijing, Beijing 100083, China | | NI Jie | School of Mechanical Engineering, University of Science and Technology Beijing, Beijing 100083, China | | JIANG Chang-hua | China Astronaut Research and Training Center Beijing, Beijing 404023, China |
|
| 摘要: |
|
| 关键词: |
| DOI：10.19554/j.cnki.1001-3563.2022.04.008 |
| 分类号: |
| 基金项目:This research is supported by the Open Funding Project of National Key Laboratory of Human Factors Engineering (Grant NO. 6142222190309). The authors acknowledged the kindness of MAHNOB-HCI, who provided the inducing materials for this study. |
|
| Multimodal Emotion Recognition Based on Facial Expression and ECG Signal |
|
NIU Jian-wei1, AN Yue-qi1, NI Jie1, JIANG Chang-hua2
|
|
(1.School of Mechanical Engineering, University of Science and Technology Beijing, Beijing 100083, China;2.China Astronaut Research and Training Center Beijing, Beijing 404023, China)
|
| Abstract: |
| As a key link in human-computer interaction, emotion recognition can enable robots to correctly perceive user emotions and provide dynamic and adjustable services according to the emotional needs of different users, which is the key to improve the cognitive level of robot service. Emotion recognition based on facial expression and electrocardiogram has numerous industrial applications. First, three-dimensional convolutional neural network deep learning architecture is utilized to extract the spatial and temporal features from facial expression video data and electrocardiogram (ECG) data, and emotion classification is carried out. Then two modalities are fused in the data level and the decision level, respectively, and the emotion recognition results are then given. Finally, the emotion recognition results of single-modality and multi-modality are compared and analyzed. Through the comparative analysis of the experimental results of single-modality and multi-modality under the two fusion methods, it is concluded that the accuracy rate of multi-modal emotion recognition is greatly improved compared with that of single-modal emotion recognition, and decision-level fusion is easier to operate and more effective than data-level fusion. |
| Key words: multi-modal emotion recognition facial expression ECG signal three-dimensional convolutional neural network |
渝公网安备 50010702501716号