# Adapting Large Language Models via Reading Comprehension

**Authors:** Daixuan Cheng, Shaohan Huang, Furu Wei  
**arXiv:** [2309.09530](https://arxiv.org/abs/2309.09530)  
**Date:** 2023-09-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.09530v1/

## Key points

- Authors Cheng, Huang, and Wei explore the impact of continued pre-training on domain-specific corpora on large language models.
- Training on raw corpora imparts domain knowledge but hampers question-answering ability; authors propose converting raw corpora into reading comprehension texts.
- Enriching raw text with tasks related to content boosts performance in biomedicine, finance, and law domains.
- Their 7B language model achieves competitive results comparable to larger-scale domain-specific models like BloombergGPT-50B.
- Domain-specific reading comprehension texts enhance model performance even on general benchmarks.
- The authors provide access to their model, code, and data for further exploration and implementation at https://github.com/microsoft/LMOps.

## Summary

In their paper titled "Adapting Large Language Models via Reading Comprehension," authors Daixuan Cheng, Shaohan Huang, and Furu Wei delve into the impact of continued pre-training on domain-specific corpora on large language models. They discover that while training on raw corpora imparts domain knowledge to the model, it significantly hampers its ability to answer questions effectively. Drawing inspiration from human learning processes, particularly reading comprehension where practice enhances question-answering skills based on acquired knowledge, the authors propose a straightforward method for converting raw corpora into reading comprehension texts. This method involves enriching each raw text with a series of tasks related to its content. Their approach proves to be highly scalable and applicable across various pre-training corpora, consistently boosting performance in biomedicine, finance, and law domains. Notably, their 7B language model achieves competitive results comparable to much larger-scale domain-specific models like BloombergGPT-50B. Additionally, they demonstrate that utilizing domain-specific reading comprehension texts can enhance the model's performance even on general benchmarks, hinting at the potential for developing a versatile model spanning multiple domains. The authors provide access to their model, code, and data through https://github.com/microsoft/LMOps for further exploration and implementation. Through their research findings and innovative methodology, they pave the way for leveraging reading comprehension techniques to adapt large language models effectively across diverse domains.