# Olmo 3

**Authors:** Team Olmo,  :, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh  
**arXiv:** [2512.13961](https://arxiv.org/abs/2512.13961)  
**Date:** 2025-12-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2512.13961v1/

## Key points

- Olmo 3 is a family of state-of-the-art and fully-open language models available at the 7B and 32B parameter scales.
- The construction of Olmo 3 models focuses on enhancing long-context reasoning, function calling, coding, instruction following, general chat, and knowledge recall capabilities.
- The flagship model within this family is the Olmo 3 Think 32B, known as the most robust fully-open thinking model currently released.
- A long-context recipe was developed using documents sourced from the olmOCR science PDFs pool to extend the context capabilities of Olmo 3.
- The Dolma 3 Longmino Pool combines 34% long-context data with 66% high-quality short-context data sampled from Dolma 3 Dolmino Mix for training purposes.
- Various techniques were applied during the long-context extension process including utilizing YaRN in full attention layers and implementing document packing and inter-document masking.
- Performance evaluation of context-extended models was conducted on two popular long-context benchmarks: RULER and HELMET.
- Sourcing of long context data primarily involved scientific PDFs processed by olmOCR and filtered based on gzip compressibility metrics.
- Refinement process included experiments to determine optimal components for enhancing performance in long-context scenarios.
- Applying YaRN to full attention layers yielded superior outcomes while leveraging olmOCR science PDFs proved more effective compared to alternative methods.

## Summary

Olmo 3 is a family of state-of-the-art and fully-open language models available at the 7B and 32B parameter scales. The construction of the Olmo 3 models focuses on enhancing long-context reasoning, function calling, coding, instruction following, general chat, and knowledge recall capabilities. The flagship model within this family is the Olmo 3 Think 32B which stands out as the most robust fully-open thinking model currently released. To extend the context capabilities of Olmo 3, a long-context recipe was developed using documents sourced from the olmOCR science PDFs pool. This collection known as Dolma 3 Longmino Pool combines 34% long-context data with 66% high-quality short-context data sampled from Dolma 3 Dolmino Mix. Training on this mix involved an additional 7B and 32B models. Various techniques were applied during the long-context extension process including utilizing YaRN in full attention layers and implementing document packing and inter-document masking. The performance of the context-extended models was evaluated on two popular long-context benchmarks: RULER and HELMET. RULER consists of synthetic long-context tasks like Needle-in-a-Haystack variations and aggregation tasks serving as a primary metric for guiding recipe development. On the other hand HELMET offers a diverse set of long-context benchmarks covering retrieval in-context learning and summarization tasks to assess more general capabilities. The sourcing of long context data primarily involved scientific PDFs processed by olmOCR and filtered based on gzip compressibility metrics. Additional filtering considerations were made using LongPpl to identify key tokens requiring long-range dependencies. The refinement process also included experiments to determine optimal components for enhancing performance in long-context scenarios. Overall results indicated that applying YaRN to full attention layers yielded superior outcomes while leveraging olmOCR science PDFs proved more effective compared to alternative methods. Synthetic data augmentation further enhanced performance over natural documents alone. Document packing was identified as a crucial factor in boosting performance for longer contexts. In conclusion through meticulous analysis and experimentation with architectural design decisions outlined in Bertsch et al., Olmo 3 has successfully expanded its capabilities in handling complex long-context tasks with impressive results showcased across various benchmarks.