# Enabling Large Language Models to Generate Text with Citations

**Authors:** Tianyu Gao, Howard Yen, Jiatong Yu, Danqi Chen  
**arXiv:** [2305.14627](https://arxiv.org/abs/2305.14627)  
**Date:** 2023-05-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14627v1/

## Key points

- Authors address the issue of hallucination in generated outputs of large language models (LLMs)
- Importance of improving factual correctness and verifiability in text generation by enabling LLMs to include citations
- Proposal of ALCE benchmark for Automatic LLMs' Citation Evaluation
- ALCE includes diverse set of questions and retrieval corpora, requiring end-to-end systems for evidence retrieval and citation inclusion
- Introduction of automatic metrics for fluency, correctness, and citation quality that correlate with human judgments
- Experiment results show significant room for improvement in current systems, with instances where 49% of generated text lacks complete citation support
- Potential future directions include developing better retrievers and improving information synthesis from multiple sources
- Contribution to enhancing reliability and accuracy of text generated by LLMs through proper citation integration
- ALCE benchmark as a crucial tool for evaluating and comparing different approaches in automated citation generation within large language models

## Summary

In their paper titled "Enabling Large Language Models to Generate Text with Citations," authors Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen address the issue of hallucination in generated outputs of large language models (LLMs). They emphasize the importance of improving factual correctness and verifiability in text generation by enabling LLMs to include citations. The existing methods for evaluating citation quality often rely on commercial search engines and human evaluation, which can be challenging to reproduce and compare across different modeling approaches. To address this gap, the authors propose ALCE, a benchmark designed specifically for Automatic LLMs' Citation Evaluation. , , , , 

ALCE includes a diverse set of questions and retrieval corpora, requiring the development of end-to-end systems that can retrieve supporting evidence and generate answers with proper citations. The benchmark introduces automatic metrics along three dimensions - fluency, correctness, and citation quality - which have shown strong correlation with human judgments. Through experiments involving state-of-the-art LLMs and innovative prompting strategies, the authors demonstrate that current systems have significant room for improvement. For instance, on the ELI5 dataset, even the best model exhibits instances where 49% of its generated text lacks complete citation support. Further analysis highlights potential future directions for enhancing these systems, such as developing better retrievers,,and improving the synthesis of information from multiple sources. Overall,this research contributes valuable insights into enhancing the reliability and accuracy of text generated by LLMs through proper citation integration. The ALCE benchmark serves as a crucial tool for evaluating and comparing different approaches in this domain while paving the way for advancements in automated citation generation within large language models.