# All NLP Tasks Are Generation Tasks: A General Pretraining Framework

**Authors:** Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang  
**arXiv:** [2103.10360](https://arxiv.org/abs/2103.10360)  
**Date:** 2021-03-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.10360v1/

## Key points

- The paper introduces a novel pretraining architecture called GLM (General Language Model) to address limitations of existing NLP frameworks
- GLM performs exceptionally well on classification, unconditional generation, and conditional generation tasks using a single pretrained model
- GLM outperforms BERT-like models in classification tasks due to improved pretrain-finetune consistency
- GLM naturally handles variable-length blank filling crucial for many downstream tasks
- Empirical results demonstrate GLM's superiority over BERT on the SuperGLUE natural language understanding benchmark with the same amount of pre-training data
- GLM achieves best performance in natural language understanding (NLU), conditional generation, and unconditional generation simultaneously compared to BERT-Large with 1.25x parameters
- Technical details include dividing input into Part A and Part B for efficient processing by GLM's transformer with masked self-attention mechanism, and autoregressive generation of Part B spans through query keys and self-attention masks

## Summary

The paper "All NLP Tasks Are Generation Tasks: A General Pretraining Framework" introduces a novel pretraining architecture called GLM (General Language Model) to address the limitations of existing pretraining frameworks in natural language processing (NLP). The current landscape of pretraining architectures includes autoregressive models like GPT, autoencoding models like BERT, and encoder-decoder models like T5. However, these frameworks do not excel across all NLP tasks, which complicates model development and selection. <br>
 is a rapidly evolving field with various tasks such as classification, unconditional generation, and conditional generation. To tackle the challenges posed by these diverse tasks, GLM offers several key advantages over previous models. Firstly, it performs exceptionally well on classification, unconditional generation, and conditional generation tasks using a single pretrained model. This  sets it apart from other frameworks. Secondly, GLM outperforms BERT-like models in classification tasks due to improved pretrain-finetune consistency. Lastly, GLM naturally handles variable-length blank filling - a crucial aspect for many downstream tasks.<br>
Empirical results demonstrate the superiority of GLM over BERT on the SuperGLUE natural language understanding benchmark with the same amount of pre-training data. Additionally,<br>
when compared to BERT-Large with 1.25x parameters,<br>
GLM achieves the best performance in natural language understanding (NLU), conditional generation,<br>
and unconditional generation simultaneously.<br>
This showcases its  and effectiveness across various NLP tasks.<br>
Furthermore,<br>
the paper delves into technical details such as dividing input into Part A and Part B for efficient processing by GLM's transformer with masked self-attention mechanism. It also explains how Part B spans are generated autoregressively through query keys and self-attention masks.<br>
In conclusion, "All NLP Tasks Are Generation Tasks: A General Pretraining Framework" presents a groundbreaking approach in NLP research with GLM's ability to excel in diverse NLP tasks through its innovative architecture and superior performance metrics compared to existing models.