Improved Baselines with Visual Instruction Tuning

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning

   Federico Cocchi, Nicholas Moratelli, Davide Caffagni et al. · 2025 · 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · cited 37 times · Not summarized yet

   Abstract

   Recent progress in Multimodal Large Language Models (MLLMs) has highlighted the critical roles of both the visual backbone and the underlying language model. While prior work has primarily focused on scaling these components to billions of parameters, the trade-offs between model size, architecture, and performance remain under-explored. Additionally, inconsistencies in training data and evaluation protocols have hindered direct comparisons, making it difficult to derive optimal design choices. In this paper, we introduce LLaVA-MORE, a new family of MLLMs that integrates recent language models with diverse visual backbones. To ensure fair comparisons, we employ a unified training protocol ap…

2. Very close Not relevant

   LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training

   Xiang An, Yin Xie, Kai-Cheng Yang et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 274 times · Not summarized yet

   Abstract

   We present LLaVA-OneVision-1.5, a novel family of Large Multimodal Models (LMMs) that achieve state-of-the-art performance with significantly reduced computational and financial costs. Different from the existing works, LLaVA-OneVision-1.5 provides an open, efficient, and reproducible framework for building high-quality vision-language models entirely from scratch. The LLaVA-OneVision-1.5 release comprises three primary components: (1) Large-Scale Curated Datasets: We construct an 85M concept-balanced pretraining dataset LLaVA-OneVision-1.5-Mid-Traning and a meticulously curated 22M instruction dataset LLaVA-OneVision-1.5-Instruct. (2) Efficient Training Framework: We develop a complete end-…

3. Very close Not relevant

   Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning

   Vedanshu, M. Tripathi, Bhavnesh Jaint · 2024 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   Abstract

   The integration of large language models (LLMs) with vision-language (VL) tasks has been a transformative development in the realm of artificial intelligence, highlighting the potential of LLMs as a versatile general-purpose chatbot. However, the current trend in this evolution focuses on the integration of vision and language to create models that can operate in more diverse and real-world contexts. We present a novel approach, termed Bottleneck Adapter, specifically crafted for enhancing the multimodal functionalities of these complex models, enabling joint optimization of the entire multimodal LLM framework through a process known as Multimodal Model Tuning (MMT). Our approach utilizes li…

4. Very close Groundwork for this paper Not relevant

   Visual Instruction Tuning

   Haotian Liu, Chunyuan Li, Qingyang Wu et al. · 2023 · Computer Vision and Pattern Recognition · cited 11,800 times · AI summary available

   Visual Instruction Tuning introduces LLaVA, whose vision-language connector and instruction-tuning approach this paper modifies to establish stronger baselines.

   Abstract

   Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the beha…

5. Very close Not relevant

   An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models

   Ya-Dong Lu, Chunyuan Li, Haotian Liu et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 44 times · Not summarized yet

   Abstract

   Visual instruction tuning has recently shown encouraging progress with open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However, most existing studies of open-source LMM are performed using models with 13B parameters or smaller. In this paper we present an empirical study of scaling LLaVA up to 33B and 65B/70B, and share our findings from our explorations in image resolution, data mixing and parameter-efficient training methods such as LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language capabilities when completing real-world tasks in the wild. We find that scaling LMM consistently enhances model performance and improves language capabilities…

6. Very close Not relevant

   TinyLLaVA: A Framework of Small-scale Large Multimodal Models

   Baichuan Zhou, Ying Hu, Xi Weng et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 213 times · Not summarized yet

   Abstract

   We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects of different vision encoders, connection modules, language models, training data and training recipes. Our extensive experiments showed that better quality of data combined with better training recipes, smaller LMMs can consistently achieve on-par performances compared to bigger LMMs. Under our framework, we train a family of small-scale LMMs. Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as baselines for f…

7. Very close Another approach to the problem Not relevant

   Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

   Daechul Ahn, Yura Choi, Youngjae Yu et al. · 2024 · Computer Vision and Pattern Recognition · cited 44 times · AI summary available

   Tuning Large Multimodal Models for Videos uses reinforcement learning from AI feedback to align video multimodal models, whereas this paper improves LLaVA baselines with an MLP connector and VQA data.

   Abstract

   Recent advancements in large language models have influenced the development of video large multimodal models (VLMMs). The previous approaches for VLMMs involved Supervised Fine-Tuning (SFT) with instruction-tuned datasets, integrating LLM with visual encoders, and adding additional learnable modules. Video and text multimodal alignment remains challenging, primarily due to the deficient volume and quality of multimodal instruction-tune data compared to text-only data. We present a novel alignment strategy that employs multimodal AI system to oversee itself called Reinforcement Learning from AI Feedback (RLAIF), providing self-preference feedback to refine itself and facilitating the alignme…

8. Very close Same method, another problem Cites this paper Not relevant

   An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

   Liang Chen, Haozhe Zhao, Tianyu Liu et al. · 2024 · Computer Vision and Pattern Recognition · cited 158 times · AI summary available

   FastV targets LLaVA-based vision-language models, applying visual-token pruning to accelerate inference rather than improving their training baselines.

   Abstract

   In this study, we identify the inefficient attention phenomena in Large Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5, QwenVL-Chat and Video-LLaVA. We find out that the attention computation over visual tokens is of extreme inefficiency in the deep layers of popular LVLMs, suggesting a need for a sparser approach compared to textual data handling. To this end, we introduce FastV, a versatile plug-and-play method designed to optimize computational efficiency by learning adaptive attention patterns in early layers and pruning visual tokens in subsequent ones. Our evaluations demonstrate FastV's ability to dramatically reduce computational costs (e.g., a 45 redu…

Papers that cite it

1. Very close Not relevant

   Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

   Zhe Chen, Jiannan Wu, Wenhai Wang et al. · 2024 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cited 736 times · Not summarized yet

   Abstract

   The exponential growth of large language models (LLMs) has opened up numerous possibilities for multi-modal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs. In this work, we design a large-scale vision-language foun-dation model (Intern VL), which scales up the vision foun-dation model to 6 billion parameters and progressively aligns it with the LLM, using web-scale image-text data from various sources. This model can be broadly applied to and achieve state-of-the-art performance on 32 generic visual-linguistic benchmarks including visual perception tasks such as image-level o…

2. Close Not relevant

   A Survey on Multimodal Large Language Models

   Shukang Yin, Chaoyou Fu, Sirui Zhao et al. · 2023 · Computer Vision and Pattern Recognition · cited 897 times · AI summary available

   Abstract

   Multimodal Large Language Model (MLLM) recently has been a new rising research hotspot, which uses powerful Large Language Models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of MLLM, such as writing stories based on images and OCR-free math reasoning, are rare in traditional methods, suggesting a potential path to artificial general intelligence. In this paper, we aim to trace and summarize the recent progress of MLLM. First of all, we present the formulation of MLLM and delineate its related concepts. Then, we discuss the key techniques and applications, including Multimodal Instruction Tuning (M-IT), Multimodal In-Context Learning (M-ICL), Multimodal…

3. Close Not relevant

   VILA: On Pre-training for Visual Language Models

   Ji Lin, Hongxu Yin, Wei Ping et al. · 2024 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cited 250 times · Not summarized yet

   Abstract

   Visual language models (VLMs) rapidly progressed with the recent success of large language models. There have been growing efforts on visual instruction tuning to extend the LLM with visual inputs, but lacks an in-depth study of the visual language pre-training process, where the model learns to perform joint modeling on both modalities. In this work, we examine the design options for VLM pre-training by augmenting LLM towards VLM through step-by-step controllable comparisons. We introduce three main findings: (1) freezing LLMs during pre-training can achieve decent zero-shot performance, but lack in-context learning capability, which requires unfreezing the LLM; (2) interleaved pre-training…

4. Close Uses this paper in depth Not relevant

   ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering

   A. Senthil · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Visual Question Answering (VQA), yet they continue to struggle with questions requiring precise spatial reasoning and fine-grained visual understanding. These limitations often manifest as object, attribute, and spatial hallucinations, where models generate confident but visually unsupported responses due to insufficient region-level and fine-grained visual grounding. To address this challenge, we propose ReVA, a region-aware VQA model that employs a frozen CLIP ViT-L/14 Vision Transformer (ViT) and a Qwen2.5-7B-Instruct large language model (LLM) connected through a dual bridge that aligns both whole-image and reg…

5. Close Not relevant

   Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

   Sicong Leng, Hang Zhang, Guanzheng Chen et al. · 2024 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cited 199 times · Not summarized yet

   Abstract

   Large Vision-Language Models (LVLMs) have advanced considerably, intertwining visual recognition and language understanding to generate content that is not only coherent but also contextually attuned. Despite their success, LVLMs still suffer from the issue of object hallucinations, where models generate plausible yet incorrect outputs that include objects that do not exist in the images. To mitigate this issue, we introduce Visual Contrastive Decoding (VCD), a simple and training-free method that contrasts output distributions derived from original and distorted visual inputs. The proposed VCD effectively reduces the over-reliance on statistical bias and unimodal priors, two essential cause…