# Textbooks Are All You Need II: phi-1.5 technical report

**Authors:** Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, Yin Tat Lee  
**arXiv:** [2309.05463](https://arxiv.org/abs/2309.05463)  
**Date:** 2023-09-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.05463v1/

## Key points

- Investigation into the power of smaller Transformer-based language models
- Building upon the success of previous models such as TinyStories and phi-1
- Exploring how small a Large Language Model (LLM) can be while still achieving high levels of capability
- Questions about the necessity of large-scale models like GPT-4
- Cost, energy consumption, and controllability concerns with large models
- Tackling the challenging task of common sense reasoning in natural language
- Introduction of phi-1.5, a 1.3 billion parameter model trained on a dataset of 30 billion tokens
- Comparable benchmark results in common sense reasoning to larger models trained on larger datasets
- Dataset consists almost exclusively of synthetically generated data
- Synthetic data generation implications for controlling toxic and biased content generation with LLMs

## Summary

In this work, we continue the investigation into the power of smaller Transformer-based language models. We build upon the success of previous models such as TinyStories and phi-1, which demonstrated the ability to generate coherent English and achieve state-of-the-art Python coding performance, respectively. Our goal is to explore how small a Large Language Model (LLM) can be while still achieving high levels of capability. The latest generation of LLMs, exemplified by GPT-4, has shown remarkable improvements over their predecessors. However, these models are extremely large in scale, with trillions of parameters and tokens. This raises important questions about the necessity of such scale for achieving advanced capabilities. The cost of training and maintaining large models is substantial, and concerns about energy consumption and controllability also come into play. By investigating whether similar capabilities can be achieved at a smaller scale, we aim to provide insights into intelligent system architectures and development. Previous research has focused on tasks like fluent English speaking and coding simple functions in Python. In this work, we tackle the challenging task of common sense reasoning in natural language. Common sense reasoning has long been a difficult task for AI systems. We introduce phi-1.5, a new 1.3 billion parameter model trained on a dataset of 30 billion tokens. Remarkably, phi-1.5 achieves benchmark results in common sense reasoning that are comparable to models ten times its size trained on datasets more than ten times larger. One key aspect of our approach is that our dataset consists almost exclusively of synthetically generated data. This follows the methodology proposed in previous work on coding tasks using LLMs. Synthetic data generation has important implications for controlling toxic and biased content generation with LLMs—a notorious challenge in this field.