# AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

**Authors:** Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, Xipeng Qiu  
**arXiv:** [2402.12226](https://arxiv.org/abs/2402.12226)  
**Date:** 2024-02-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.12226v1/

## Key points

- Developed AnyGPT: an any-to-any multimodal language model integrating speech, text, images, and music using discrete representations
- Does not require alterations to architecture or training paradigms; relies on data-level preprocessing for new modalities
- Curated a multimodal text-centric dataset and synthesized a large-scale any-to-any multimodal instruction dataset
- Two-stage approach: generating text-based conversations with multimodal elements and constructing scenarios based on user inputs related to games and interactive media
- Leveraged LLaMA-2 for fine-tuning model responses across all modalities
- Experimental results show comparable performance to specialized models in facilitating any-to-any multimodal conversation
- Demonstrated effectiveness of discrete representations in unifying multiple modalities within the language model

## Summary

In order to address the limitations of existing multimodal language models, we have developed AnyGPT - an innovative any-to-any multimodal language model that seamlessly integrates various modalities such as speech, text, images, and music using discrete representations. Unlike traditional large language models (LLMs), AnyGPT does not require alterations to its architecture or training paradigms. Instead, it relies on data-level preprocessing to incorporate new modalities similar to adding new languages. To facilitate this integration, we have curated a multimodal text-centric dataset for alignment pre-training and synthesized a large-scale any-to-any multimodal instruction dataset consisting of 108k samples of multi-turn conversations. Our meticulous approach involves two stages: first, generating text-based conversations with multimodal elements using GPT-4 by expanding meta topics into specific scenarios and demonstrating diverse modality combinations; secondly, constructing scenarios based on user inputs related to games and interactive media and incorporating images and music through detailed textual representations. This ensures high-quality data at scale and guides the model in synthesizing contextually appropriate conversational scenarios. Furthermore, we have leveraged LLaMA-2 for fine-tuning the model's responses to improve performance across all modalities. Experimental results demonstrate that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving comparable performance to specialized models. The demos showcased on the project website highlight the effectiveness of discrete representations in unifying multiple modalities within a language model. For more details on our methodology and results, please refer to the provided link: https://junzhan2000.github.io/AnyGPT.github.io/.