# How Do Large Language Models Acquire Factual Knowledge During Pretraining?

**Authors:** Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, Minjoon Seo  
**arXiv:** [2406.11813](https://arxiv.org/abs/2406.11813)  
**Date:** 2024-06-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.11813v3/

## Key points

- Study focuses on understanding how Large Language Models (LLMs) acquire factual knowledge during pretraining
- Pretraining on more data does not significantly improve model's ability to acquire and retain factual knowledge
- Power-law relationship between training steps and forgetting of memorization and generalization of factual knowledge
- LLMs trained with duplicated data exhibit faster forgetting, while those trained with deduplicated data show enhanced robustness against forgetting
- Deduplication enhances model performance by slowing down the forgetting of generalizing acquired factual knowledge
- Presenting models with duplicated texts too frequently can lead to a preference for generating memorized contexts over generalizing factual knowledge
- Importance of understanding how LLMs acquire factual knowledge to comprehend their behavior better
- Scaling LLMs consistently improves performance rather than accelerating acquisition of factual knowledge during pretraining
- LLMs may struggle to acquire unpopular knowledge due to insufficient exposure within shorter intervals than the learnability threshold

## Summary

This study focuses on understanding how Large Language Models (LLMs) acquire factual knowledge during pretraining. The research reveals key insights into the dynamics of factual knowledge acquisition in LLMs and introduces methods, datasets, and metrics for a detailed analysis. Contrary to expectations, pretraining on more data does not significantly improve the model's ability to acquire and retain factual knowledge. Instead, there is a power-law relationship between training steps and the forgetting of memorization and generalization of factual knowledge. LLMs trained with duplicated data exhibit faster forgetting while those trained with deduplicated data show enhanced robustness against forgetting. The study also delves into why deduplication enhances model performance by slowing down the forgetting of generalizing acquired factual knowledge. It is found that presenting models with duplicated texts too frequently can lead to a preference for generating memorized contexts over generalizing factual knowledge. Furthermore, this research emphasizes the importance of understanding how LLMs acquire factual knowledge to comprehend their behavior better. By studying underexplored behaviors of LLMs, it suggests that scaling LLMs consistently improves performance rather than accelerating the acquisition of factual knowledge during pretraining. Additionally, it highlights that LLMs may struggle to acquire unpopular knowledge due to insufficient exposure within shorter intervals than the learnability threshold. In summary, this study provides valuable insights into the dynamics of factual knowledge acquisition in large language models and opens up avenues for future research in optimizing pretraining strategies for enhancing retention of such crucial information.