# LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

**Authors:** Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li  
**arXiv:** [2408.07055](https://arxiv.org/abs/2408.07055)  
**Date:** 2024-08-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.07055v1/

## Key points

- The paper addresses limitations of current long context large language models (LLMs) in generating outputs exceeding 2,000 words.
- The authors introduce AgentWrite, an agent-based pipeline that breaks down ultra-long generation tasks into subtasks to enable off-the-shelf LLMs to produce coherent outputs surpassing 20,000 words.
- The researchers create the LongWriter-6k dataset comprising 6,000 supervised fine-tuning (SFT) data with output lengths ranging from 2k to 32k words to extend the output length of existing models to over 10,000 words while maintaining high quality.
- They develop LongBench-Write as a benchmark for evaluating ultra-long generation capabilities and achieve state-of-the-art performance through DPO enhancements.
- The study demonstrates how innovative approaches like AgentWrite can unlock the capability of existing LLMs for generating outputs exceeding 10,000 words.

## Summary

, , , , 

The paper "LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs" by authors Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li addresses the limitations of current long context large language models (LLMs) in generating outputs exceeding 2,000 words. Despite being able to process inputs up to 100,000 tokens, existing models struggle with producing longer outputs due to the lack of sufficient training data. To overcome this challenge and unlock the potential of long context LLMs for larger output windows, the authors introduce AgentWrite - an agent-based pipeline that breaks down ultra-long generation tasks into subtasks. This approach enables off-the-shelf LLMs to produce coherent outputs surpassing 20,000 words. Leveraging AgentWrite, the researchers create the LongWriter-6k dataset comprising 6,000 supervised fine-tuning (SFT) data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, they successfully extend the output length of existing models to over 10,000 words while maintaining high output quality. Furthermore, the team develops LongBench-Write as a comprehensive benchmark for evaluating ultra-long generation capabilities. Their model enhanced through DPO achieves state-of-the-art performance on this benchmark and outperforms even larger proprietary models. In conclusion,<nl>
the study demonstrates how innovative approaches like AgentWrite can unlock the capability of existing LLMs to generate outputs exceeding 10,000 words. The findings underscore the importance of dataset composition in enhancing model performance and highlight avenues for further advancements in natural language processing tasks requiring extensive text generation. Interested readers can access the code and models related to this study on GitHub at https://github.com/THUDM/LongWriter.