# A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks

**Authors:** Xuanfan Ni, Piji Li  
**arXiv:** [2405.10251](https://arxiv.org/abs/2405.10251)  
**Date:** 2024-05-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.10251v1/

## Key points

- Researchers evaluated large language models (LLMs) in natural language generation (NLG) tasks
- Evaluated models included ChatGPT, ChatGLM, T5-based models, LLaMA-based models, and Pythia-based models
- Used EmpatheticDialogue dataset for Dialogue Generation and CNN/DailyMail dataset along with XSum dataset for Text Summarization
- Employed LCCC dataset for Chinese dialogue generation and THUCNews along with LCSTS for Chinese text summarization
- Highlighted capabilities of Transformer-based LLMs with billions of parameters in understanding natural language and solving complex tasks
- Noted few-shot learning properties of these models allowing them to perform new tasks based on textual instructions or minimal examples
- Discussed trends in scaling these models further to enhance their capabilities

## Summary

, , , , 

In this study, the researchers conducted a comprehensive evaluation of large language models (LLMs) in the context of natural language generation (NLG) tasks. The evaluation included well-known and high-performing LLMs such as ChatGPT, ChatGLM, T5-based models, LLaMA-based models, and Pythia-based models. The English dataset used for Dialogue Generation was EmpatheticDialogue, which consists of 25k empathetic conversations between a speaker and a listener. For Text Summarization, the researchers utilized the CNN/DailyMail dataset containing 93k and 220k articles from CNN and Daily Mail websites respectively, along with the XSum dataset which includes extreme summarization examples from BBC articles. Furthermore, for Chinese NLG tasks, the researchers employed the LCCC dataset for dialogue generation and THUCNews along with LCSTS for text summarization. The evaluation settings incorporated input templates and post-processing strategies to provide a common ground for comparison across different LLMs. The study reported both automatic results and detailed analysis to assess the performance of these models in NLG tasks. The paper highlighted that large language models (LLMs), particularly Transformer-based models with billions of parameters trained on extensive text corpora, have shown significant capabilities in understanding natural language and solving complex tasks. These models have demonstrated few-shot learning properties where they can perform new tasks based on textual instructions or minimal examples. The research also discussed trends in scaling these models further to enhance their capabilities. In conclusion, the comprehensive assessment of various LLMs in NLG tasks revealed notable trends and phenomena that shed light on their multilingual capabilities. The results from both automatic evaluations and manual analyses provided valuable insights into the performance of these models across different languages and datasets.