# SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models

**Authors:** Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, Shreyas Saxena  
**arXiv:** [2303.10464](https://arxiv.org/abs/2303.10464)  
**Date:** 2023-03-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.10464v1/

## Key points

- The pre-training and fine-tuning paradigm has revolutionized Natural Language Processing (NLP)
- Pre-training on large datasets before fine-tuning on task-specific data is computationally expensive
- Sparse Pre-training and Dense Fine-tuning (SPDF) approach uses unstructured weight sparsity to train only a subset of weights during pre-training
- Recover the representational capacity by allowing zeroed weights to learn during dense fine-tuning
- SPDF effectiveness evaluated on natural language generation (NLG) tasks such as E2E, WebNLG, and DART and text summarization tasks such as Curation Corpus using GPT-2 Small and GPT-3 XL models
- High degrees of weight sparsity can be induced during pre-training without significant degradation in accuracy across all NLG tasks
- Performance of sparse pre-trained model is correlated with difficulty of fine-tuning task
- As size of model increases, it becomes more amenable to higher sparsity levels
- GPT models can be pre-trained with 50%-75% sparsity without losing significant accuracy on downstream tasks
- Fully sparse end-to-end training can prevent models from generalizing well on downstream tasks
- Transitioning from sparse to dense matrices during fine-tuning mitigates poor generalizability due to sparse only training.
- SPDF presents a promising direction for training large GPT models at a fraction of the training FLOPs while retaining benefits for downstream NLP tasks.
- Study establishes relationship between sparsity, task complexity, dataset size and provides insights into how best to use SPDF for efficient training of large scale language models.

## Summary

The pre-training and fine-tuning paradigm has revolutionized Natural Language Processing (NLP) by enabling language models to learn cross-domain knowledge through pre-training on large datasets before fine-tuning on task-specific data. However, the computational costs of pre-training are prohibitively high, requiring orders of magnitude more FLOPs than fine-tuning. To address this challenge, a new approach called Sparse Pre-training and Dense Fine-tuning (SPDF) has been proposed. This approach uses unstructured weight sparsity to train only a subset of weights during pre-training and then recovers the representational capacity by allowing the zeroed weights to learn during dense fine-tuning. The effectiveness of SPDF has been evaluated on natural language generation (NLG) tasks such as E2E, WebNLG, and DART and text summarization tasks such as Curation Corpus using GPT-2 Small and GPT-3 XL models. The study validates three hypotheses: first, high degrees of weight sparsity can be induced during pre-training without significant degradation in accuracy across all NLG tasks; second, the performance of the sparse pre-trained model is correlated with the difficulty of the fine-tuning task; third, as the size of the model increases, it becomes more amenable to higher sparsity levels. The results indicate that these GPT models can be pre-trained with 50%-75% sparsity without losing significant accuracy on downstream tasks. However, fully sparse end-to-end training can prevent models from generalizing well on downstream tasks. Therefore transitioning from sparse to dense matrices during fine-tuning mitigates poor generalizability due to sparse only training. Overall, SPDF presents a promising direction for training large GPT models at a fraction of the training FLOPs while retaining benefits for downstream NLP tasks. The study establishes a relationship between sparsity, task complexity, dataset size and provides insights into how best to use SPDF for efficient training of large scale language models.