# Improved Baselines with Visual Instruction Tuning

**Authors:** Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee  
**arXiv:** [2310.03744](https://arxiv.org/abs/2310.03744)  
**Date:** 2023-10-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.03744v1/

## Key points

- The paper "Improved Baselines with Visual Instruction Tuning" presents advancements in large multimodal models (LMM) within the LLaVA framework.
- Implementation of a fully-connected vision-language cross-modal connector and incorporation of academic task-related data, such as Visual Question Answering (VQA) prompts, lead to state-of-the-art performance across 11 benchmarks.
- Modifications in LLaVA framework are powerful and data-efficient, resulting in stronger baselines with high training sample efficiency.
- LLaVA's simplicity stands out compared to other approaches like InstructBLIP or Qwen-VL, which rely on complex visual resamplers trained on massive amounts of image-text paired data.
- By training only a simple fully-connected projection layer on a relatively small dataset of 600K image-text pairs, impressive results are achieved in approximately one day on a single 8-A100 machine.
- LLaVA's approach differs from Qwen-VL by exclusively using publicly available data for training purposes, enhancing reproducibility and accessibility in state-of-the-art LMM research.
- Improvements made to the LLaVA framework showcase its potential for advancing multimodal understanding capabilities efficiently and effectively.

## Summary

The paper "Improved Baselines with Visual Instruction Tuning" presents advancements in large multimodal models (LMM) within the LLaVA framework. The authors demonstrate that by implementing a fully-connected vision-language cross-modal connector and incorporating academic task-related data, such as Visual Question Answering (VQA) prompts, they achieve state-of-the-art performance across 11 benchmarks. These modifications are shown to be powerful and data-efficient, leading to stronger baselines with high training sample efficiency. Compared to other approaches like InstructBLIP or Qwen-VL, which rely on complex visual resamplers trained on massive amounts of image-text paired data, LLaVA stands out for its simplicity. By utilizing a straightforward architecture design and training only a simple fully-connected projection layer on a relatively small dataset of 600K image-text pairs, the authors achieve impressive results. The final model can be trained in approximately one day on a single 8-A100 machine. Notably, LLaVA's approach differs from Qwen-VL by exclusively using publicly available data for training purposes. This decision enhances reproducibility and accessibility in state-of-the-art LMM research. The improvements made to the LLaVA framework showcase its potential for advancing multimodal understanding capabilities efficiently and effectively. Overall, the study highlights the significance of simple yet impactful modifications in enhancing multimodal models' performance and making cutting-edge research more accessible and reproducible within the field of large multimodal models.