# Language Models are General-Purpose Interfaces

**Authors:** Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, Furu Wei  
**arXiv:** [2206.06336](https://arxiv.org/abs/2206.06336)  
**Date:** 2022-06-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.06336v1/

## Key points

- Authors propose using language models as universal interfaces for interacting with diverse foundation models
- Methodology involves pretrained encoders for different modalities integrated with a language model as a universal task layer
- Introduce semi-causal language modeling objective for joint pretraining of interface and encoders
- Approach combines elements of causal and non-causal modeling to create a versatile framework
- Advantages include leveraging bidirectional encoders for effective fine-tuning while retaining benefits of in-context learning and open-ended generation
- Capabilities unlocked include in-context learning, instruction following, and finetuning with encoders
- Experiments across multiple benchmarks show outperformance or favorable competition with specialized models in tasks like finetuning, zero-shot generalization, and few-shot learning

## Summary

In their paper titled "Language Models are General-Purpose Interfaces," authors Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei delve into the realm of foundation models and their widespread applicability in various downstream tasks. The authors propose a novel approach using language models as universal interfaces to interact with diverse foundation models. Their methodology involves a collection of pretrained encoders that can perceive different modalities such as vision and language. These encoders are then integrated with a language model acting as a universal task layer. To facilitate this integration, the authors introduce a semi-causal language modeling objective that enables joint pretraining of the interface and modular encoders. By combining elements from both causal and non-causal modeling approaches, their method harnesses the strengths of each to create a versatile framework. One key advantage of their approach is its ability to leverage bidirectional encoders for effective fine-tuning while retaining the benefits of in-context learning and open-ended generation from causal language modeling. This seamless combination unlocks various capabilities such as in-context learning and instruction following with finetuned encoders. The authors conducted experiments across multiple language-only and vision-language benchmarks to evaluate the performance of their model. The results demonstrate that their proposed approach outperforms or competes favorably with specialized models in tasks such as finetuning, zero-shot generalization, and few-shot learning. With 32 pages dedicated to detailing their research findings, this work represents a significant contribution towards enhancing the adaptability and effectiveness of foundation models through the use of language models as general-purpose interfaces.