# Evaluating Large Language Models on Controlled Generation Tasks

**Authors:** Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Frederick Wieting, Nanyun Peng, Xuezhe Ma  
**arXiv:** [2310.14542](https://arxiv.org/abs/2310.14542)  
**Date:** 2023-10-23  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2310.14542v1/

## Key points

- Authors evaluate controllability of large language models (LLMs) on various generation tasks
- LLMs compared to smaller specialized models
- Performance analyzed on five tasks and ten benchmarks
- LLMs struggle with fine-grained hard constraints like numerical planning and paraphrase generation
- LLMs can generate human-level rationales and conform to coarse control signals like sentiment, topic, and keyword incorporation
- Automatic rationales generated by LLMs can enhance performance through chain-of-thought reasoning
- Study has limitations including heavy prompt engineering effort and imperfect automatic evaluations
- No solutions proposed for addressing tasks where LLMs struggle, future work needed
- Research provides insights into controllability of large language models in generation tasks
- Potential solutions offered to improve performance.

## Summary

In this study, the authors evaluate the controllability of large language models (LLMs) on various generation tasks. They compare LLMs to smaller specialized models and analyze their performance on five tasks and ten benchmarks, including a numerical planning benchmark that is challenging for LLMs but easy for humans. The results show that while LLMs can generate human-level rationales and conform to coarse control signals like sentiment, topic, and keyword incorporation, they struggle with fine-grained hard constraints such as numerical planning and paraphrase generation. The authors suggest that these findings can guide the adoption of LLMs in downstream applications. They propose using automatic rationales generated by LLMs to enhance their performance through chain-of-thought reasoning. However, the study has some limitations, including heavy prompt engineering effort and imperfect automatic evaluations. Additionally, no solutions are proposed for addressing the tasks where LLMs struggle, leaving it as future work. Overall, this research provides valuable insights into the controllability of large language models in generation tasks and offers potential solutions to improve their performance.