# TextMI: Textualize Multimodal Information for Integrating Non-verbal Cues in Pre-trained Language Models

**Authors:** Md Kamrul Hasan, Md Saiful Islam, Sangwu Lee, Wasifur Rahman, Iftekhar Naim, Mohammed Ibrahim Khan, Ehsan Hoque  
**arXiv:** [2303.15430](https://arxiv.org/abs/2303.15430)  
**Date:** 2023-03-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.15430v2/

## Key points

- Pre-trained language models like BERT have achieved groundbreaking performance in natural language processing tasks.
- However, these models cannot be used for multimodal behavior understanding tasks that involve non-verbal features such as acoustic and visual cues unless they are integrated with language.
- The University of Rochester proposed TextMI, which converts acoustic and visual information into corresponding textual descriptions and concatenates them with spoken text to feed into a pre-trained BERT model.
- TextMI significantly reduces model complexity while adding interpretability to the decision-making process compared to established baseline multimodal models that use intricate fusion mechanisms.
- TextMI achieved superior performance in multimodal sarcasm detection and demonstrated better/near state-of-the-art performances in multimodal sentiment analysis and multimodal humor detection tasks.
- The pre-trained language model performs better with acoustic and visual information than textual information only.
- Compared to other baseline models, TextMI offers significant advantages by reducing computational complexity while improving interpretability.
- TextMI provides a general competitive baseline for multimodal behavioral analysis tasks particularly in a low resource setting.

## Summary

The use of pre-trained large language models such as BERT has revolutionized the field of natural language processing, achieving groundbreaking performance in various language understanding tasks. However, these models cannot be applied to multimodal behavior understanding tasks that involve non-verbal features such as acoustic and visual cues unless they are integrated with language. To address this challenge, researchers from the University of Rochester proposed TextMI, a novel approach that converts acoustic and visual information into corresponding textual descriptions and concatenates them with spoken text. The augmented input is then fed into a pre-trained BERT model and fine-tuned on three downstream multimodal tasks: sentiment analysis, humor detection and sarcasm detection. Compared to established baseline multimodal models that use intricate fusion mechanisms, TextMI significantly reduces model complexity while adding interpretability to the model's decision-making process. The study found that pre-trained language models can efficiently integrate non-verbal cues if they are described in textual form. TextMI achieved superior performance in multimodal sarcasm detection and demonstrated better/near state-of-the-art performances in multimodal sentiment analysis and multimodal humor detection tasks. Furthermore, an ablation study indicated that the pre-trained language model performs better with acoustic and visual information than textual information only. Compared to other baseline models such as TFN, LMF, MFM, ICCN, MulT, MISA and MAG-BERT that learn the multi-modal representation of text, acoustic and visual using different methods; TextMI offers significant advantages by reducing computational complexity while improving interpretability. The approach can be particularly useful for scenarios where a large dataset is scarce or interpretability is requisite. In conclusion, TextMI provides a general competitive baseline for multimodal behavioral analysis tasks particularly in a low resource setting. The approach significantly reduces model complexity and can be applied to a diverse set of tasks while achieving superior or near state of the art performance.