# OPT: Open Pre-trained Transformer Language Models

**Authors:** Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, Luke Zettlemoyer  
**arXiv:** [2205.01068](https://arxiv.org/abs/2205.01068)  
**Date:** 2022-05-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.01068v1/

## Key points

- Open Pre-trained Transformers (OPT): a suite of decoder-only pre-trained transformers with varying parameters
- Aimed at researchers and come with a non-commercial license
- Focus on understanding limitations of large language models (LLMs) before commercial deployment
- Highlight ethical and social risks associated with deploying LLMs at scale
- Emphasize the need for responsible development
- Discuss significant compute and carbon costs involved in reproducing models of this size
- OPT-175B model achieves comparable performance to GPT-3 with 1/7th of the carbon footprint to develop
- Logbook released detailing infrastructure challenges and impact on carbon emissions throughout LLM development lifecycle
- Importance of considering model training, experimentation, and downstream inference costs when measuring environmental impact
- Set of baselines provided across various scales to enable researchers to study impact and limitations of these models based on scale
- Many LLMs may have been under-trained due to limited data, suggesting incorporating more data and continuing training could improve performance further
- Evidence indicating step-function changes in capabilities occurring at smaller scales than 175B, emphasizing the need for examining a wider range of scales for different research applications
- Related work includes document deduplication using MinhashLSH, tokenization using GPT-2 byte-level BPE tokenizer, and details about corpora used such as RoBERTa corpus subsets (BookCorpus, Stories) and Pile subsets (CommonCrawl, DM Mathematics, Project Gutenberg, HackerNews, OpenSubtitles, OpenWebText2, USPTO , Wikipedia)

## Summary

In this work, the authors present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers with varying parameters. These models are aimed at researchers and come with a non-commercial license to focus on understanding the limitations of large language models (LLMs) before commercial deployment. The authors highlight the ethical and social risks associated with deploying LLMs at scale and emphasize the need for responsible development. The authors also discuss the significant compute and carbon costs involved in reproducing models of this size. They compare their OPT-175B model to GPT-3 and show that OPT-175B achieves comparable performance while requiring only 1/7th of the carbon footprint to develop. By releasing their logbook detailing infrastructure challenges, they aim to shed light on the entire LLM development lifecycle's impact on carbon emissions. They stress the importance of considering not just model training but also experimentation and downstream inference costs when measuring the environmental impact. Additionally, by providing a set of baselines across various scales, the authors hope to enable researchers to study the impact and limitations of these models solely based on scale. They mention that many LLMs may have been under-trained due to limited training data, suggesting that incorporating more data and continuing training could further improve performance. The authors also acknowledge evidence indicating step-function changes in capabilities occurring at smaller scales than 175B, emphasizing the need for examining a wider range of scales for different research applications. The paper includes related work discussing document deduplication using MinhashLSH, tokenization using GPT-2 byte-level BPE tokenizer, and details about corpora used such as RoBERTa corpus subsets (BookCorpus, Stories) and Pile subsets (CommonCrawl, DM Mathematics, Project Gutenberg, HackerNews, OpenSubtitles, OpenWebText2, USPTO , Wikipedia). They mention eliminating certain subsets from Pile due to instabilities or unsuitability. Overall, this work presents OPT as a suite of pre-trained transformers for researchers which emphasizes ethical considerations around large language models (LLMs), discusses compute and carbon costs involved in developing such models while providing baselines for studying how scale impacts model performance.