# Multimodal Chain-of-Thought Reasoning in Language Models

**Authors:** Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, Alex Smola  
**arXiv:** [2302.00923](https://arxiv.org/abs/2302.00923)  
**Date:** 2023-02-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.00923v4/

## Key points

- The paper introduces Multimodal-CoT, a novel approach that enhances the performance of large language models (LLMs) on complex reasoning tasks.
- LLMs have previously used chain-of-thought (CoT) prompting for generating intermediate reasoning chains, but only in the language modality.
- The proposed framework incorporates both language and vision modalities, separating rationale generation and answer inference stages.
- The Multimodal-CoT model outperforms the previous state-of-the-art LLM by 16 percentage points in accuracy on the ScienceQA benchmark and surpasses human performance.
- The paper discusses the benefits of joint modeling of text and visual information in improving knowledge acquisition.
- Experimental results demonstrate superior performance compared to existing approaches.
- Publicly accessible code is provided for implementing Multimodal-CoT for further exploration and development.
- This research presents a promising approach for enhancing reasoning capabilities in language models by incorporating multimodal information while using fewer parameters.

## Summary

The paper titled "Multimodal Chain-of-Thought Reasoning in Language Models" introduces a novel approach called Multimodal-CoT that enhances the performance of large language models (LLMs) on complex reasoning tasks. LLMs have previously demonstrated impressive performance by utilizing chain-of-thought (CoT) prompting to generate intermediate reasoning chains as rationales for answer inference. However, existing CoT studies have focused solely on the language modality. To address this limitation, the authors propose a two-stage framework that incorporates both language (text) and vision (images) modalities. This framework separates rationale generation and answer inference, allowing the latter to leverage better generated rationales based on multimodal information. The proposed Multimodal-CoT model, with less than 1 billion parameters, outperforms the previous state-of-the-art LLM (GPT-3.5) by 16 percentage points in terms of accuracy on the ScienceQA benchmark and even surpasses human performance. The paper provides an overview of the Multimodal-CoT task and discusses its benefits in improving knowledge acquisition through joint modeling of text and visual information. It also presents experimental results demonstrating the superior performance of their model compared to existing approaches. Furthermore, it makes available publicly accessible code for implementing Multimodal-CoT for further exploration and development. Overall, this research introduces a promising approach for enhancing reasoning capabilities in language models by incorporating multimodal information and separating rationale generation from answer inference stages. The proposed model achieves remarkable results on complex reasoning tasks while using fewer parameters than existing methods which highlights its potential as an efficient solution for natural language processing applications requiring advanced reasoning capabilities.