# Language Is Not All You Need: Aligning Perception with Language Models

**Authors:** Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei  
**arXiv:** [2302.14045](https://arxiv.org/abs/2302.14045)  
**Date:** 2023-02-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.14045v1/

## Key points

- Convergence of language, multimodal perception, action and world modeling is crucial in the quest for artificial general intelligence.
- Researchers have introduced Kosmos-1 - a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot).
- The model was trained from scratch on web-scale multimodal corpora consisting of arbitrarily interleaved text and images; image-caption pairs; and text data.
- Experimental results demonstrate that Kosmos-1 achieves impressive performance on language understanding and generation; OCR-free NLP (directly fed with document images); perception-language tasks such as multimodal dialogue, image captioning and visual question answering; and vision tasks such as image recognition with descriptions specifying classification via text instructions.
- MLLMs can benefit from cross-modal transfer by transferring knowledge from language to multimodal and vice versa.
- To further evaluate the nonverbal reasoning capability of MLLMs, the researchers also introduce a dataset of Raven IQ tests.
- The work was carried out by Shaohan Huang et al., including Li Dong, Wenhui Wang, Yaru Hao among others.

## Summary

In the quest for artificial general intelligence, a convergence of language, multimodal perception, action and world modeling is crucial. To this end, researchers have introduced Kosmos-1 – a Multimodal Large Language Model (MLLM) – that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). The model was trained from scratch on web-scale multimodal corpora consisting of arbitrarily interleaved text and images; image-caption pairs; and text data. Various settings were evaluated without any gradient updates or finetuning. Experimental results demonstrate that Kosmos-1 achieves impressive performance on language understanding and generation; OCR-free NLP (directly fed with document images); perception-language tasks such as multimodal dialogue, image captioning and visual question answering; and vision tasks such as image recognition with descriptions specifying classification via text instructions. Additionally, MLLMs can benefit from cross-modal transfer by transferring knowledge from language to multimodal and vice versa. To further evaluate the nonverbal reasoning capability of MLLMs, the researchers also introduce a dataset of Raven IQ tests. The work was carried out by Shaohan Huang et al., including Li Dong, Wenhui Wang, Yaru Hao among others.