# Language Is Not All You Need: Aligning Perception with Language Models

**Authors:** Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei  
**arXiv:** [2302.14045](https://arxiv.org/abs/2302.14045)  
**Date:** 2023-02-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.14045v2/

## Key points

- In the quest for artificial general intelligence, language, multimodal perception, action and world modeling must converge.
- A team of researchers has introduced Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot).
- The model was trained from scratch on web-scale multimodal corpora that included arbitrarily interleaved text and images, image-caption pairs and text data.
- Experimental results showed impressive performance on language understanding and generation tasks such as OCR-free NLP directly fed with document images as well as perception-language tasks such as multimodal dialogue and visual question answering.
- Vision tasks such as image recognition with descriptions were performed by specifying classification via text instructions.
- MLLMs can benefit from cross-modal transfer by transferring knowledge from language to multimodal and vice versa.
- The researchers also introduced a dataset of Raven IQ tests that diagnose the nonverbal reasoning capability of MLLMs.
- Kosmos-1 represents a significant step towards achieving artificial general intelligence through the convergence of language and multimodal perception.

## Summary

In the quest for artificial general intelligence, language, multimodal perception, action and world modeling must converge. To this end, a team of researchers has introduced Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). The model was trained from scratch on web-scale multimodal corpora that included arbitrarily interleaved text and images, image-caption pairs and text data. The team evaluated various settings without any gradient updates or finetuning. Experimental results showed impressive performance on language understanding and generation tasks such as OCR-free NLP directly fed with document images as well as perception-language tasks such as multimodal dialogue and visual question answering. Additionally, vision tasks such as image recognition with descriptions were performed by specifying classification via text instructions. The researchers also demonstrated that MLLMs can benefit from cross-modal transfer by transferring knowledge from language to multimodal and vice versa. They also introduced a dataset of Raven IQ tests that diagnose the nonverbal reasoning capability of MLLMs. Overall, Kosmos-1 represents a significant step towards achieving artificial general intelligence through the convergence of language and multimodal perception.