# LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents

**Authors:** Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, Chunyuan Li  
**arXiv:** [2311.05437](https://arxiv.org/abs/2311.05437)  
**Date:** 2023-11-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.05437v1/

## Key points

- LLaVA-Plus is a general-purpose multimodal assistant that enhances large multimodal models.
- It utilizes a skill repository of pre-trained vision and vision-language models.
- The assistant is trained on multimodal instruction-following data.
- LLaVA-Plus surpasses its predecessor, LLaVA, in existing capabilities and exhibits new ones.
- It actively engages with image queries during human-AI interaction sessions, improving tool use performance and enabling new scenarios.
- Building general-purpose multimodal assistants for computer vision and vision-language tasks is an open area of exploration.
- Development of multimodal agents can be categorized into end-to-end training with Large Language Models (LLMs) and building Large Multimodal Models (LMMs).
- End-to-end training methods have proven effective in helping LMMs gain emergent abilities in visual understanding and reasoning.
- Challenges include hallucinations and tool use conflicts in developing general-purpose multimodal AI agents.
- Researchers will publicly release comprehensive assets including generated multimodal instruction data, codebase, checkpoints, and a visual chat demo for reproducibility.
- Transparency is emphasized through detailed explanations of training data collection and model training processes.
- The paper provides a table showcasing the skill repository of LLaVA Plus along with dataset statistics for each tool use case.
- Expanded visual understanding skills include open set detection, grounding, semantic/instance/interactive segmentation, tagging captioning OCRs, and their compositions.

## Summary

LLaVA-Plus is a general-purpose multimodal assistant that enhances the capabilities of large multimodal models. It utilizes a skill repository of pre-trained vision and vision-language models to activate relevant tools based on user inputs and complete real-world tasks. The assistant is trained on multimodal instruction-following data, allowing it to acquire skills in visual understanding, generation, external knowledge retrieval, and compositions. Empirical results demonstrate that LLaVA-Plus surpasses its predecessor, LLaVA, in existing capabilities while also exhibiting new ones. Notably, LLaVA-Plus actively engages with image queries throughout human-AI interaction sessions, significantly improving tool use performance and enabling new scenarios. The introduction highlights the aspiration of developing general-purpose assistants capable of following users' multimodal instructions for various real-world tasks. While there has been progress in using Large Language Models (LLMs) for natural language tasks, building general-purpose multimodal assistants for computer vision and vision-language tasks remains an open area of exploration. The development of multimodal agents can be categorized into two classes: end-to-end training with LLMs and building Large Multimodal Models (LMMs). End-to-end training methods have proven effective in helping LMMs gain emergent abilities in visual understanding and reasoning. Notable examples include Flamingo and multimodal GPT4. However, there is still room for improvement in developing general purpose multimodal AI agents due to challenges such as hallucinations and tool use conflicts. To ensure reproducibility, the researchers will publicly release a comprehensive set of assets including generated multimodal instruction data, codebase, LLaVA Plus checkpoints, and a visual chat demo. Transparency is emphasized through detailed explanations of training data collection and model training processes. The paper also provides a table showcasing the skill repository of LLaVA Plus along with dataset statistics for each tool use case. The expanded visual understanding skills include open set detection and grounding, semantic/instance/interactive segmentation, tagging captioning OCRs and their compositions. These skills are categorized based on whether additional function arguments are required. Overall, LLaVA Plus represents a significant advancement in the development of general purpose multimodal assistants. It outperforms its predecessor in various benchmarks and demonstrates emergent multimodal interaction capabilities.