# How to Bridge the Gap between Modalities: A Comprehensive Survey on Multimodal Large Language Model

**Authors:** Shezheng Song, Xiaopeng Li, Shasha Li  
**arXiv:** [2311.07594](https://arxiv.org/abs/2311.07594)  
**Date:** 2023-11-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.07594v1/

## Key points

- Multimodal Large Language Models (MLLMs) are designed to handle diverse forms of data such as text and vision by incorporating Large Language Models (LLMs) like GPT-4.
- MLLMs have the capability to generate image narratives and respond to image-based queries, making them promising for realistic human-computer interactions and advancing artificial general intelligence.
- Despite their potential, MLLMs face challenges in processing the semantic gap in multimodal data, which can lead to erroneous outputs with societal risks.
- Selecting appropriate modality alignment methods is crucial for MLLMs to avoid increasing parameters without significant performance improvements.
- The paper aims to explore various modal alignment methods for LLMs and assess their current capabilities to address environmental concerns and improve user accessibility.
- Existing modal alignment methods are categorized into four groups: Multimodal Converters, Multimodal Perceivers, Tools Assistance, and Data-Driven methods.
- The field of multimodal information alignment is still evolving, with ongoing research efforts aimed at enhancing these processes within MLLMs.

## Summary

In their paper titled "How to Bridge the Gap between Modalities: A Comprehensive Survey on Multimodal Large Language Model," authors Shezheng Song, Xiaopeng Li, and Shasha Li delve into the realm of Multimodal Large Language Models (MLLMs). These models are designed to handle diverse forms of data such as text and vision by incorporating Large Language Models (LLMs) like GPT-4. The authors highlight the impressive capabilities of MLLMs, including their ability to generate image narratives and respond to image-based queries. This makes MLLMs a promising avenue for more realistic human-computer interactions and potentially advancing artificial general intelligence. Despite their promising features, MLLMs still face obstacles in effectively processing the semantic gap inherent in multimodal data. This challenge can result in erroneous outputs, posing potential risks to society. Therefore, the authors emphasize the critical importance of selecting appropriate modality alignment methods for MLLMs. Improper techniques may lead to an increase in parameters without significant performance enhancements. The primary objective of this paper is to explore various modal alignment methods for LLMs and assess their current capabilities. By implementing effective modality alignment strategies, LLMs can better address environmental concerns and improve accessibility for users. The study categorizes existing modal alignment methods into four groups: Multimodal Converters that adapt data for LLM comprehension; Multimodal Perceivers that enhance how LLMs interpret different data types; Tools Assistance for converting data into a unified format, typically text; and Data-Driven methods that educate LLMs on understanding specific data types within a dataset. The field of multimodal information alignment is still in its nascent stages of exploration and experimentation. Moving forward, the authors aim to organize and update a range of research methodologies aimed at enhancing multimodal information alignment processes within MLLMs. Through continued research efforts in this area, advancements can be made towards improving the efficiency and accuracy of these models in handling diverse forms of data effectively.