# Visual Instruction Tuning

**Authors:** Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee  
**arXiv:** [2304.08485](https://arxiv.org/abs/2304.08485)  
**Date:** 2023-04-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.08485v2/

## Key points

- Use of language-only GPT-4 to generate multimodal language-image instruction-following data
- Introduction of LLaVA (Large Language and Vision Assistant) as an end-to-end trained large multimodal model
- Collection of 158K unique language-image instruction-following samples, including conversations, detailed descriptions, and complex reasoning questions
- Impressive multimodal chat abilities demonstrated by LLaVA with a relative score of 85.1% compared to GPT-4 on a synthetic dataset
- State-of-the-art accuracy of 92.53% achieved when LLaVA combined with GPT-4 is fine-tuned on Science QA
- Availability of GPT-4 generated visual instruction tuning data, model, and code base publicly
- Detailed process of generating detailed descriptions and complex reasoning questions for images described
- Architecture of LLaVA illustrated, leveraging the capabilities of both pre-trained LLM and visual model
- Training details provided, including organization of multi-turn conversation data and instruction-tuning using auto-regressive training objective
- Performance evaluation on challenging tasks using LLaVA-Bench (In-the-Wild) dataset where it outperforms other models in terms of accuracy on complex reasoning questions
- Acknowledgment of limitations regarding weaknesses revealed by the challenging benchmark dataset

## Summary

This paper explores the use of language-only GPT-4 to generate multimodal language-image instruction-following data, with the aim of improving zero-shot capabilities on new tasks in the multimodal field. The authors introduce LLaVA (Large Language and Vision Assistant), an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding. They collect a total of 158K unique language-image instruction-following samples, including conversations, detailed descriptions, and complex reasoning questions. The experiments show that LLaVA demonstrates impressive multimodal chat abilities and achieves a relative score of 85.1% compared to GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, LLaVA combined with GPT-4 achieves a new state-of-the-art accuracy of 92.53%. The authors also provide GPT-4 generated visual instruction tuning data, as well as their model and code base publicly available. Additionally, they describe in detail the process of generating detailed descriptions and complex reasoning questions for images. The architecture of LLaVA is illustrated, which effectively leverages the capabilities of both the pre-trained LLM and visual model. Training details are provided, including how multi-turn conversation data is organized and how instruction-tuning is performed using the original auto-regressive training objective. The paper also presents an evaluation of LLaVA's performance in more challenging tasks using LLaVA-Bench (In-the-Wild) dataset, where it outperforms other models such as BLIP-2 and OpenFlamingo in terms of accuracy on complex reasoning questions. However, limitations are acknowledged regarding weaknesses revealed by this challenging benchmark dataset. Overall, this work contributes to advancing multimodal models by incorporating language-only GPT-4 for generating multimodal language-image instruction-following data and achieving improved performance on various tasks through fine-tuning.