# Pretrained Transformers as Universal Computation Engines

**Authors:** Kevin Lu, Aditya Grover, Pieter Abbeel, Igor Mordatch  
**arXiv:** [2103.05247](https://arxiv.org/abs/2103.05247)  
**Date:** 2021-03-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.05247v1/

## Key points

- Study investigates pretraining transformer architecture on natural language for transfer learning to other modalities
- Introduces Frozen Pretrained Transformer (FPT) model that finetunes certain layers while leaving self-attention layers untouched
- Goal is to enhance performance and efficiency in non-language tasks like numerical computation, vision, and protein fold prediction
- Transformers are powerful alternative to RNNs for sequence processing due to self-attention mechanisms
- Pretraining self-attention layers on natural language data aims to leverage feature representations for transfer learning without extensive finetuning
- Experiments with pretrained GPT-2 model on various tasks across different modalities show promising results with FPT model achieving comparable performance by finetuning a small fraction of parameters
- FPT models converge faster during training, suggesting learned self-attention layers facilitate efficient universal computation
- Investigates pretraining regimes, architecture choices, attention maps, generalization abilities, model sizes, and parameter importance in transformers
- Findings highlight potential of leveraging pretrained language models for zero-shot generalization to diverse modalities with sequential structures

## Summary

In this study, we investigate the potential of pretraining a transformer architecture on natural language for transfer learning to other modalities with minimal finetuning. We introduce a Frozen Pretrained Transformer (FPT) model that only finetunes certain layers while leaving the self-attention layers untouched. Our goal is to explore how pretraining on natural language can enhance performance and efficiency in non-language tasks such as numerical computation, vision, and protein fold prediction. Traditionally, recurrent neural networks (RNNs) have been used for sequence processing tasks. However, transformers have emerged as a powerful alternative due to their ability to extract features across tokens using self-attention mechanisms. By pretraining the self-attention layers on data-rich natural language data, we aim to leverage the learned feature representations for transfer learning to different modalities without extensive finetuning. We conduct experiments with a pretrained GPT-2 model on various tasks spanning different modalities. The FPT model shows promising results by achieving comparable performance to fully trained models while only finetuning a small fraction of parameters. Furthermore, FPT models converge faster during training, suggesting that the self-attention layers learned from language pretraining may facilitate efficient universal computation. Through our investigations, we delve into pretraining regimes, architecture choices, attention maps, generalization abilities, model sizes and parameter importance in transformers. Our findings highlight the potential of leveraging pretrained language models for zero-shot generalization to diverse modalities with sequential structures.