# OLMo: Accelerating the Science of Language Models

**Authors:** Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot  
**arXiv:** [2402.00838](https://arxiv.org/abs/2402.00838)  
**Date:** 2024-02-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.00838v1/

## Key points

- OLMo is a state-of-the-art and truly open language model
- It provides access to the entire framework for building and studying language modeling
- Unlike previous efforts, OLMo releases powerful, open language models instead of just model weights and inference code
- OLMo can be used to study biases and potential risks in language models
- Future plans include releasing training logs, ablations, findings, adaptation models, code, and data
- Various teammates and collaborators contributed to different aspects of OLMo's development
- Considerations are highlighted when training language models on different data sources like curated sources compared to scraped web text
- Variations in performance across evaluation sources based on similarities between training and evaluation distributions are discussed
- Large-scale language model considerations are briefly mentioned
- The goal is to continuously support and extend OLMo's capabilities by incorporating different model sizes, modalities, datasets, safety measures, and evaluations into the framework

## Summary

This technical report introduces OLMo, a state-of-the-art and truly open language model. It provides access to the entire framework for building and studying the science of language modeling. Unlike previous efforts that only released model weights and inference code, OLMo empowers the research community by providing them with powerful, open language models. These can be used to study biases and potential risks. The report also mentions future plans to release training logs, ablations, findings, and adaptation models along with their code and data. The authors acknowledge the contributions of various teammates and collaborators in different aspects of OLMo's development. This includes individuals involved in pretraining dataset construction and tooling, model training and architecture, evaluation suite development, model adaptation, license creation, and risk assessment. is highlighted as an important consideration when training language models on heterogeneous data sources like curated sources (e.g., Wikipedia) compared to scraped web text. The report also discusses variations in performance across different evaluation sources based on similarities between training and evaluation distributions. Additionally,and  considerations associated with large-scale language models are briefly mentioned. In conclusion,to continuously support and extend OLMo's capabilities by incorporating different model sizes, modalities, datasets, safety measures,and evaluations into the framework. They hope that this release will not only strengthen the open research community but also inspire new innovations in language modeling.