# Chameleon: Mixed-Modal Early-Fusion Foundation Models

**Authors:**  Chameleon Team  
**arXiv:** [2405.09818](https://arxiv.org/abs/2405.09818)  
**Date:** 2024-05-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.09818v1/

## Key points

- Chameleon is a family of early-fusion token-based mixed-modal models designed for understanding and generating images and text in any sequence.
- Chameleon excels in tasks such as visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation.
- The key to Chameleon's success lies in its fully token-based architecture, allowing seamless integration of information across modalities.
- Chameleon outperforms competitors like Llama-2 in text-only tasks and remains competitive with models such as Mixtral 8x7B and Gemini-Pro.
- The stable training approach of Chameleon ensures consistent performance across various vision-language benchmarks.

## Summary

, , , , 


In this paper, the authors introduce Chameleon, a family of early-fusion token-based mixed-modal models that excel in understanding and generating images and text in any arbitrary sequence. The models are designed with a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. Chameleon is evaluated on a wide range of tasks including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. <kw>Chameleon:</kw> A versatile model for understanding and generating images and text. The results show that Chameleon demonstrates broad and general capabilities with state-of-the-art performance in image captioning tasks. It outperforms Llama-2 in text-only tasks while remaining competitive with models such as Mixtral 8x7B and Gemini-Pro. Furthermore, Chameleon showcases non-trivial image generation capabilities within a single model. It matches or exceeds the performance of much larger models like Gemini Pro and GPT-4V according to human judgments on a new long-form mixed-modal generation evaluation. <kw>Early-Fusion Token-Based Mixed-Modal Models:</kw> A powerful approach for multimodal reasoning. The key to Chameleon's success lies in its fully token-based architecture which allows for seamless information integration across modalities. By quantizing images into discrete tokens and training on mixed-modal data from scratch, Chameleon learns to jointly reason over image and text in a unique way that late-fusion architectures cannot achieve. <kw>Stable Training Approach:</kw> Ensuring consistent performance across various vision-language benchmarks. Overall, Chameleon marks a significant advancement in unified modeling of full multimodal documents by providing strong performance across various vision-language benchmarks while enabling new mixed-modal reasoning and generation capabilities. <kw>Mixed-Modal Setting:</kw> A challenging but crucial aspect of multimodal modeling. <kw>Fully Token-Based Architecture:</kw> Enabling seamless integration of information from different modalities.