# CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

**Authors:** Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, Edouard Grave  
**arXiv:** [1911.00359](https://arxiv.org/abs/1911.00359)  
**Date:** 2019-11-01  
**Summary page:** https://www.summarizepaper.com/en/paper/1911.00359v1/

## Key points

- Pre-training text representations has become popular in NLP and led to improvements in various NLP tasks
- Quality of pre-training models depends on size and quality of pretraining corpora
- "CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data" paper describes an automatic pipeline for extracting high-quality monolingual datasets from Common Crawl for various languages
- Pipeline follows data processing method introduced in fastText by Mikolov et al. (2017) and Grave et al. (2018)
- Pipeline is augmented with filtering step to select documents close to high-quality corpora like Wikipedia
- Common Crawl provides vast amount of web crawl data that can be used for training NLP models, but quality varies significantly across different websites and languages
- Extracting high-quality monolingual datasets from Common Crawl can greatly benefit NLP research
- This approach can help improve the performance of pre-trained text representations in many NLP applications

## Summary

In recent years, pre-training text representations has become a popular approach in natural language processing (NLP) that has led to significant improvements in various NLP tasks. However, the quality of these models heavily relies on the size and quality of the pretraining corpora. In this paper titled "CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data," authors Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave describe an automatic pipeline for extracting massive high-quality monolingual datasets from Common Crawl for various languages. The authors' pipeline follows the data processing method introduced in fastText by Mikolov et al. (2017) and Grave et al. (2018), which involves deduplicating documents and identifying their language. The pipeline is augmented with a filtering step to select documents that are close to high-quality corpora like Wikipedia. The authors' approach is particularly useful because Common Crawl provides a vast amount of web crawl data that can be used for training NLP models. However, the quality of this data varies significantly across different websites and languages. Therefore, extracting high-quality monolingual datasets from Common Crawl can greatly benefit NLP research. Overall, this paper presents a valuable contribution to the field of NLP by providing an efficient way to extract high-quality monolingual datasets from web crawl data for various languages. This approach can help improve the performance of pre-trained text representations in many NLP applications.