# Extending Context Window of Large Language Models via Positional Interpolation

**Authors:** Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian  
**arXiv:** [2306.15595](https://arxiv.org/abs/2306.15595)  
**Date:** 2023-06-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.15595v1/

## Key points

- Position Interpolation (PI) is a novel method that extends the context window sizes of RoPE-based pretrained LLMs.
- PI allows for up to 32768 context window size with minimal fine-tuning and has shown strong empirical results on various tasks requiring long context.
- PI works by linearly down-scaling the input position indices to match the original context window size instead of extrapolating beyond the trained length.
- Theoretical analysis supports interpolation as a more stable alternative to extrapolation.
- PI retains the original architecture of models and can reuse existing optimization and infrastructure.
- Experiments on long document summarization using GovReport dataset show competitive ROUGE scores compared to other baselines.
- PI complements retrieval-augmented LLMs by allowing more documents to be included in the input without modifying the attention mechanism or model architecture.

## Summary

The article introduces Position Interpolation (PI), a novel method that extends the context window sizes of RoPE-based pretrained LLMs. This approach allows for up to 32768 context window size with minimal fine-tuning and has shown strong empirical results on various tasks requiring long context. The authors demonstrate its effectiveness in passkey retrieval, language modeling, and long document summarization. PI works by linearly down-scaling the input position indices to match the original context window size instead of extrapolating beyond the trained length. This prevents high attention scores that can disrupt the self-attention mechanism. Theoretical analysis also supports interpolation as a more stable alternative to extrapolation. One of the key advantages of PI is that it retains the original architecture of models and can reuse existing optimization and infrastructure. In their experiments on long document summarization using GovReport dataset, the authors fine-tune extended LLaMA models with a context window of 16384 after truncating all input documents to their first 15000 tokens. The results show competitive ROUGE scores compared to other baselines. The article also discusses related work in retrieval-augmented LLMs and highlights how PI complements these approaches by allowing more documents to be included in the input without modifying the attention mechanism or model architecture. Overall, PI provides an effective solution for extending context window sizes in RoPE-based pretrained LLMs while maintaining stability and reusability of existing models.