# A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

**Authors:** Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, Dawei Song  
**arXiv:** [2201.05337](https://arxiv.org/abs/2201.05337)  
**Date:** 2022-01-14  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2201.05337v5/

## Key points

- Controllable Text Generation (CTG) is a rapidly growing area in natural language generation (NLG)
- CTG focuses on developing advanced text generation technologies to meet specific constraints in practical applications
- One popular approach in CTG is the use of large-scale pre-trained language models (PLMs), particularly transformer-based PLMs
- Limited interpretability of deep neural networks poses challenges to ensuring controllability in these methods
- The paper presents a systematic critical review of common tasks, main approaches, and evaluation methods in CTG using transformer-based PLMs
- PPLM is highlighted as a representative method that trains an attribute discriminant model to guide the PLM in generating corresponding text based on user-specified attributes or a single learning layer
- PPLM achieves significant improvement in attribute alignment but slightly decreases text fluency measured by perplexity
- MEGATRON-CNTR combines external knowledge and PLMs for controllable story generation using predictors, retrievers, and rankers
- FAIR is proposed as a content-controlled text generation framework that utilizes BERT for constructing content plans and BART for filling masked tokens in generated text templates without modifying their structure
- Experimental results demonstrate improved relevance and coherence between key phrases and generated texts with FAIR
- Challenges faced by the field of CTG include reliance on large datasets for deep learning based methods and potential bias in handcrafted features

## Summary

Controllable Text Generation (CTG) is a rapidly growing area in natural language generation (NLG) that focuses on developing advanced text generation technologies to meet specific constraints in practical applications. One popular approach in CTG is the use of large-scale pre-trained language models (PLMs), particularly transformer-based PLMs, which allow for the generation of more diverse and fluent text. However, the limited interpretability of deep neural networks poses challenges to ensuring controllability in these methods. In this paper, the authors present a systematic critical review of common tasks, main approaches, and evaluation methods in CTG using transformer-based PLMs. They discuss various techniques that have emerged in the past 3-4 years to address different CTG tasks with different types of controlled constraints. The authors highlight PPLM as a representative method, which trains an attribute discriminant model to guide the PLM in generating corresponding text based on user-specified attributes or a single learning layer. While PPLM achieves significant improvement in attribute alignment, it slightly decreases text fluency measured by perplexity. Another notable framework discussed is MEGATRON-CNTR, which combines external knowledge and PLMs for controllable story generation. It uses a predictor to obtain keywords for the next sentence based on the story context and retrieves relevant knowledge-enhanced sentences from an external knowledge base using a knowledge retriever. A ranker then selects the most relevant sentences, which are fed into GPT-2 (a type of PLM) along with the story context to generate the next sentence. Human evaluation results show high control success rates using keywords. Additionally, FAIR is proposed as a content-controlled text generation framework that utilizes BERT for constructing content plans and BART for filling masked tokens in generated text templates without modifying their structure. An iterative refinement algorithm within sequence-to-sequence models improves generation quality through flexible editing. Experimental results demonstrate improved relevance and coherence between key phrases and generated texts. The authors also discuss the challenges faced by the field of CTG such as reliance on large datasets for deep learning based methods and potential bias in handcrafted features; highlighting emergence of PLMs as new paradigm capable of end to end learning and generating high quality text without external domain knowledge but requiring further research to address limitations and explore promising future directions.