# Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis

**Authors:** Jia Li, Ziyang Zhang, Junjie Lang, Yueqi Jiang, Liuwei An, Peng Zou, Yangyang Xu, Sheng Gao, Jie Lin, Chunxiao Fan, Xiao Sun, Meng Wang  
**arXiv:** [2208.03051](https://arxiv.org/abs/2208.03051)  
**Date:** 2022-08-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2208.03051v2/

## Key points

- Authors present solutions for the Multimodal Sentiment Analysis Challenge (MuSe) 2022
- MuSe challenge consists of three sub-challenges: MuSe-Humor, MuSe-Reaction, and MuSe-Stress
- Focus of MuSe 2022 is on humor detection, emotional reactions, and multimodal emotional stress using different modalities and datasets
- Authors extract various multimodal features including acoustic, visual text, and biological features
- Features are fused using TEMMA and GRU with self-attention mechanism frameworks
- New audio features such as facial expression features and paragraph level text embedding are extracted to improve accuracy in sentiment analysis tasks
- Mining and blending the extracted multimodal features enhance the accuracy and reliability of multimodal sentiment prediction
- Effective data augmentation strategies are applied during model training to mitigate sample imbalance issues and prevent biased learning
- Model achieves an AUC score of 0.8932 in the MuSe-Humor sub-challenge evaluation metric
- Model outperforms all other participants with a Pearson's Correlation Coefficient score of 0.3879 in the MuSe-Reaction sub-challenge evaluation on the test set
- Model surpasses baseline performance on arousal and valence metrics in the MuSe-Stress sub-challenge evaluation with a final combined result of 0.5151

## Summary

In this paper, titled "Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis," authors Jia Li, Ziyang Zhang, Junjie Lang, Yueqi Jiang, Liuwei An, Peng Zou, Yangyang Xu, Sheng Gao, Jie Lin, Chunxiao Fan, Xiao Sun and Meng Wang present their solutions for the Multimodal Sentiment Analysis Challenge (MuSe) 2022. The challenge consists of three sub-challenges: MuSe-Humor, MuSe-Reaction and MuSe-Stress. The focus of MuSe 2022 is on humor detection, emotional reactions and multimodal emotional stress using different modalities and datasets. To address these challenges in their work the authors extract various multimodal features including acoustic, visual text and biological features. These features are then fused using TEMMA and GRU with self-attention mechanism frameworks. To improve accuracy in sentiment analysis tasks the authors extract several new audio features such as facial expression features and paragraph level text embedding. They substantially enhance the accuracy and reliability of multimodal sentiment prediction by mining and blending the extracted multimodal features. Effective data augmentation strategies are applied during model training to mitigate sample imbalance issues and prevent biased learning of subject characters. For the MuSe-Humor sub-challenge evaluation metric AUC score of 0.8932 is achieved by their model. In the MuSe-Reaction sub-challenge evaluation on the test set using Pearson's Correlation Coefficient as a metric their approach outperforms all other participants with a score of 0.3879. In addition to that for the MuSe-Stress sub challenge evaluation on arousal and valence metrics they surpass the baseline performance on the test dataset with a final combined result of 0.5151. This paper is scheduled to appear in MuSe 2022 which is a workshop co located with ACM MM2022 conference . It is authored by Jia Li , Ziyang Zhang , Junjie Lang , Yueqi Jiang , Liuwei An , Peng Zou , Yangyang Xu , Sheng Gao , Jie Lin , Chunxiao Fan , Xiao Sun and Meng Wang . The paper consists of 8 pages including 2 figures .