# Emotion Recognition System from Speech and Visual Information based on Convolutional Neural Networks

**Authors:** Nicolae-Catalin Ristea, Liviu Cristian Dutu, Anamaria Radoi  
**arXiv:** [2003.00351](https://arxiv.org/abs/2003.00351)  
**Date:** 2020-02-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2003.00351v1/

## Key points

- Emotion recognition is a significant area of research in human-computer interactions
- Combining visual and audio information improves results compared to using each source separately
- Facial expressions can be analyzed to recognize human emotions through Facial Action Units
- The paper proposes a real-time emotion recognition system based on deep Convolutional Neural Networks (CNNs)
- Speech data is also analyzed and fused with visual information to enhance accuracy
- Experimental results demonstrate the effectiveness of this approach for emotion recognition
- Previous work, such as the EmotiW challenge, supports the significance of multimodal emotion recognition
- End-to-end multimodal emotion recognition using deep neural networks further supports the idea of combining multiple modalities for improved performance
- Advancements in generating realistic facial animations based on speech input are referenced, such as a study on speech-driven facial animation with Generative Adversarial Networks (GANs)
- A multi-modal sequence fusion approach for emotion recognition that combines video and audio information is mentioned.

## Summary

Emotion recognition has emerged as a significant area of research in the field of human-computer interactions. Recent advancements have shown that combining visual and audio information leads to improved results compared to using each source separately. From a visual perspective, facial expressions can be analyzed to recognize human emotions, specifically through the combination of various Facial Action Units. In this paper, the authors propose a real-time emotion recognition system based on deep Convolutional Neural Networks (CNNs) that achieves high accuracy rates. To enhance the accuracy of the recognition system, the authors also analyze speech data and fuse the information from both visual and audio sources. Experimental results demonstrate the effectiveness of this approach for emotion recognition and highlight the importance of combining visual and audio data. The paper references previous work in this field, including an Emotion Recognition in the Wild Challenge (EmotiW) challenge and workshop summary [11], which emphasizes the significance of multimodal emotion recognition. Another study [12] explores end-to-end multimodal emotion recognition using deep neural networks, further supporting the idea that combining multiple modalities improves performance. Additionally, a study on realistic speech-driven facial animation with Generative Adversarial Networks (GANs) [13] is referenced, indicating advancements in generating realistic facial animations based on speech input. The authors also mention a multi-modal sequence fusion approach for emotion recognition [14], which utilizes recursive attention to combine video and audio information. Overall, this refined summary provides more context by referencing related studies that support the importance of multimodal emotion recognition and highlighting recent advancements in generating realistic facial animations based on speech input.