# Lost in the Middle: How Language Models Use Long Contexts

**Authors:** Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang  
**arXiv:** [2307.03172](https://arxiv.org/abs/2307.03172)  
**Date:** 2023-07-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.03172v2/

## Key points

- Language models' ability to process extensive input contexts is explored
- Performance of language models on multi-document question answering and key-value retrieval tasks is analyzed
- Language model performance is highest when relevant information is at the beginning or end of the input context
- Significant degradation occurs when models need to access relevant information in the middle of long contexts
- Even explicitly long-context models experience a decrease in performance as input context grows longer
- The analysis provides insights into how language models utilize their input context
- New evaluation protocols for future long-context models are introduced
- The work contributes to understanding language model behavior and can guide advancements in natural language processing research.

## Summary

The paper titled "Lost in the Middle: How Language Models Use Long Contexts" by Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang explores how well language models utilize long contexts. Recent language models have the capability to process extensive input contexts but there is limited understanding of their effectiveness when doing so. The authors analyze the performance of these models on two tasks: multi-document question answering and key-value retrieval which require identifying relevant information within the input context. The findings reveal that language model performance is highest when relevant information is located at the beginning or end of the input context. However, a significant degradation occurs when models need to access relevant information in the middle of long contexts. Additionally, as the input context grows longer even explicitly long-context models experience a substantial decrease in performance. This analysis provides valuable insights into how language models utilize their input context and introduces new evaluation protocols for future long-context models. The authors' work contributes to a better understanding of language model behavior and can guide further advancements in natural language processing research.