# The Lottery Ticket Hypothesis for Pre-trained BERT Networks

**Authors:** Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, Michael Carbin  
**arXiv:** [2007.12223](https://arxiv.org/abs/2007.12223)  
**Date:** 2020-07-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2007.12223v2/

## Key points

- The paper explores the use of large pre-trained models like BERT as a starting point for training in NLP and other areas of deep learning.
- It delves into the lottery ticket hypothesis, which suggests that smaller subnetworks within models can be trained to full accuracy in isolation and transferred to other tasks.
- The authors assess whether such trainable, transferrable subnetworks exist in pre-trained BERT models.
- Matching subnetworks were found at 40% to 90% sparsity for a range of downstream tasks at pre-trained initialization, which is different from prior NLP research where they emerge only after some amount of training.
- Subnetworks found on the masked language modeling task transfer universally, while those found on other tasks transfer in a limited fashion if at all.
- This research demonstrates that the main lottery ticket observations remain relevant in the context of large-scale pre-training becoming an increasingly central paradigm in deep learning.
- Codes are available for further exploration and analysis at https://github.com/VITA-Group/BERT-Tickets.

## Summary

The Lottery Ticket Hypothesis for Pre-trained BERT Networks is a research paper authored by Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang and Michael Carbin. The paper explores the concept of using large pre-trained models like BERT as the standard starting point for training in natural language processing (NLP) and other areas of deep learning. It also delves into the lottery ticket hypothesis which suggests that models for NLP and computer vision contain smaller matching subnetworks capable of training to full accuracy in isolation and transferring to other tasks. The authors combine these observations to assess whether such trainable, transferrable subnetworks exist in pre-trained BERT models. They find matching subnetworks at 40% to 90% sparsity for a range of downstream tasks at pre-trained initialization. This is different from prior NLP research where they emerge only after some amount of training. The subnetworks found on the masked language modeling task (the same task used to pre-train the model) transfer universally; those found on other tasks transfer in a limited fashion if at all. As large-scale pre-training becomes an increasingly central paradigm in deep learning, this research demonstrates that the main lottery ticket observations remain relevant in this context. The authors provide codes available at https://github.com/VITA-Group/BERT-Tickets for further exploration and analysis. Overall, this study sheds light on how smaller subnetworks can be trained effectively within larger pre-trained models like BERT and their potential applications in various downstream tasks.