# Visual Instruction Tuning

**Authors:** Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee  
**arXiv:** [2304.08485](https://arxiv.org/abs/2304.08485)  
**Date:** 2023-04-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.08485v1/

## Key points

- Language-only GPT-4 used to generate multimodal language-image instruction-following data
- Introduction of LLaVA, an end-to-end trained large multimodal model connecting vision encoder and LLM for visual and language understanding
- LLaVA outperforms GPT-4 on synthetic multimodal instruction-following dataset
- LLaVA combined with GPT-4 achieves new state-of-the-art accuracy when fine-tuned on Science QA
- Suggestions for future exploration: increasing pre-training data scale, connecting other powerful vision models to enhance LLaVA's capabilities
- Experiments on ScienceQA benchmark showcase image understanding and conversation abilities of LLaVA through Chatbot demo
- Visual instruction tuning using language-only GPT 4 proves effective and opens possibilities for advancements in multimodal language image understanding.

## Summary

This paper presents the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. The authors introduce LLaVA, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding. They demonstrate that LLaVA achieves impressive multimodal chat abilities and outperforms GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, LLaVA combined with GPT-4 achieves a new state-of-the-art accuracy. The authors suggest several directions for future exploration, including increasing the scale of pre-training data and connecting other powerful vision models to enhance LLaVA's capabilities. Experiments on the ScienceQA benchmark showcase the image understanding and conversation abilities of LLaVA through a Chatbot demo. Overall, this work highlights the effectiveness of visual instruction tuning using language only GPT 4 and opens up possibilities for further advancements in multimodal language image understanding.