# Phi-4 Technical Report

**Authors:** Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim  
**arXiv:** [2412.08905](https://arxiv.org/abs/2412.08905)  
**Date:** 2024-12-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.08905v1/

## Key points

- Development of phi-4, a 14-billion parameter language model:
- Strong emphasis on data quality throughout training process
- Strategic incorporation of synthetic data at various stages
- Inclusion of reasoning-dense documents from sources like arXiv, PubMed Central, and GitHub, as well as licensed books
- Filtering approach for capturing information-rich web sources:
- Selection of high-quality documents using classifiers trained on LLM-generated annotations
- Over-indexing on STEM-related keywords with specialized pipeline to amplify non-STEM content
- Custom extraction and cleaning pipelines implemented for uniformity across heterogeneous data sources
- Supervised fine-tuning using curated user prompts and direct preference optimization based on rejection sampling and LLM evaluation
- Based on decoder-only transformer architecture with 14 billion parameters
- Emphasis on accuracy in code execution and proofs validity while fostering systematic reasoning
- Creation of synthetic data spanning 50 types of datasets to enhance pretraining and midtraining phases
- Surpassed GPT-4 in STEM-focused QA capabilities due to improved data quality, training curriculum enhancements, and innovations in post-training techniques

## Summary

In the development of phi-4, a 14-billion parameter language model, there was a strong emphasis on data quality throughout the training process. Unlike traditional language models that rely primarily on organic data sources such as web content or code for pre-training, phi-4 strategically incorporated synthetic data at various stages of training. This approach aimed for comprehensiveness and cleanliness above standard corpora by including major repositories of reasoning-dense documents from sources like arXiv, PubMed Central, and GitHub, as well as licensed books. To capture information-rich web sources beyond traditional datasets, a filtering approach was taken to select high-quality documents from web dumps using classifiers trained on LLM-generated annotations. This method over-indexed on STEM-related keywords and prompted the development of a specialized pipeline to amplify non-STEM content such as arts, history, travel, culture, and entertainment. Multilingual datasets were also integrated to ensure the model's ability to handle various languages. Custom extraction and cleaning pipelines were implemented to maintain uniformity across heterogeneous data sources by ingesting different file formats and preserving fragile content like equations and code blocks in web data. During post-training phases, supervised fine-tuning using curated user prompts and direct preference optimization based on rejection sampling and LLM evaluation were conducted. The phi-4 model is based on a decoder-only transformer architecture with 14 billion parameters. Emphasis was placed on accuracy in code execution and proofs validity while fostering systematic reasoning through chain-of-thought learning approaches. Synthetic data spanning 50 types of datasets were created using different seeds and prompting procedures to enhance pretraining and midtraining phases. Overall, phi-4's performance surpassed its teacher model GPT-4 in STEM-focused QA capabilities due to improved data quality, training curriculum enhancements, and innovations in post-training techniques. The model also demonstrated strong performance on reasoning-focused benchmarks despite minimal changes to its architecture from phi-3.