AI summary · arXiv: 2309.16039v1 · cs.CL

Effective Long-Context Scaling of Foundation Models

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta,
and 13 more authorsKarthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma
Original paper · DOI 10.18653/v1/2024.naacl-long.260 Submitted to arXiv on 27 Sep. 2023 Published in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (2024) License: CC BY 4.0

How to read this summary? Illustrated guide

The essentials

  • Long-context Large Language Models (LLMs) with effective context windows of up to 32,768 tokens
  • Continual pretraining from Llama 2 with longer training sequences and upsampling of long texts
  • Consistent improvements on most regular tasks and significant improvements on long-context tasks compared to Llama 2
  • Surpassing the overall performance of gpt-3.5-turbo-16k on long-context tasks using a cost-effective instruction tuning procedure
  • In-depth analysis of limitations in modeling long dependencies and impact of design choices in pretraining process
  • Evaluation and measurements of other existing open-sourced long-context models for more context
  • Importance of maintaining strong performance on standard short-context tasks while demonstrating effectiveness in real-world scenarios
  • Evaluation of fine-tuned LLMs on safety benchmarks and red teaming exercises for safety in long-context understanding
  • Acknowledgment of limitations such as limited functionality for certain applications and challenges related to tokenizer efficiency

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: Long-context LLMs

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
In this paper, the authors present a series of long-context Large Language Models (LLMs) that have effective context windows of up to 32,768 tokens. These models are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. The authors extensively evaluate these models on language modeling, synthetic context probing tasks, and various research benchmarks. The results show that the models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks compared to Llama 2. Notably, the 70B variant of the model surpasses the overall performance of gpt-3.5-turbo-16k on a suite of long-context tasks using a cost-effective instruction tuning procedure that does not require human-annotated long instruction data. The paper also includes an in-depth analysis of the individual components of their method. They discuss the limitations of Llama's position encodings in modeling long dependencies and examine the impact of various design choices in the pretraining process. Additionally, the authors provide more context by discussing evaluations and measurements of other existing open-sourced long-context models. They highlight the importance of maintaining strong performance on standard short-context tasks while demonstrating effectiveness in diverse real-world scenarios. Furthermore, they evaluate fine-tuned LLMs on safety benchmarks and perform red teaming exercises to ensure safety in long-context understanding. Although there is currently no open sourced safety benchmark designed for long context understanding, their internal red teaming did not observe significant risks compared to LLAMA 2 CHAT. The paper acknowledges some limitations such as limited functionality for certain applications that require long form outputs and challenges related to tokenizer efficiency. In conclusion, this paper presents a series of long context LLMs that outperform previous models on both regular and long context tasks. The authors provide detailed analyses and evaluations to support their findings while addressing potential limitations and future research directions.

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.

AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero Zotero guide
Citation alerts More than 300 papers already cite it: be emailed each new one. Citation alerts guide
Who works on it?

Among the 31 papers around this one, the name that comes back most: Wei Ping (3 papers)

7 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.

Continue with Google Create a free account: your first map is free
Weekly letter The latest letter of its field, "Artificial intelligence this week" of 16 November 2026: Artificial intelligence this week: What impressive scores leave out · Receive one by email for free

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close Cites this paper
    How to Train Long-Context Language Models (Effectively)
    Tianyu Gao, Alexander Wettig, Howard Yen et al. · 2024 · Computation and Language · cited 184 times · AI summary available
    Abstract

    We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. We first establish a reliable evaluation protocol to guide model development -- Instead of perplexity or simple needle-in-a-haystack (NIAH) tests, we use a broad set of long-context tasks, and we evaluate models after SFT with instruction data as this better reveals long-context abilities. Supported by our robust evaluations, we run thorough experiments to decide the data mix for continued pre-training, the instruction tuning dataset, and many other design choices. We find that (1) code repositories and books are excellent sources of long data, but it is cr…

  2. Very close
    From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
    Che-Jian Xu, Wei Ping, Peng Xu et al. · 2025 · Annual Meeting of the Association for Computational Linguistics · cited 20 times · Not summarized yet
    Abstract

    Long-context capabilities are essential for a wide range of applications, including document and video understanding, in-context learning, and inference-time scaling, all of which require models to process and reason over long sequences of text and multimodal data. In this work, we introduce a efficient training recipe for building ultra-long context LLMs from aligned instruct model, pushing the boundaries of context lengths from 128K to 1M, 2M, and 4M tokens. Our approach leverages efficient continued pretraining strategies to extend the context window and employs effective instruction tuning to maintain the instruction-following and reasoning abilities. Our UltraLong-8B, built on Llama3.1-…

  3. Very close Cites this paper
    Training-Free Long-Context Scaling of Large Language Models
    Chen An, Fei Huang, Jun Zhang et al. · 2024 · International Conference on Machine Learning · cited 76 times · Not summarized yet
    Abstract

    The ability of Large Language Models (LLMs) to process and generate coherent text is markedly weakened when the number of input tokens exceeds their pretraining length. Given the expensive overhead of finetuning large-scale models with longer sequences, we propose Dual Chunk Attention (DCA), which enables Llama2 70B to support context windows of more than 100k tokens without continual training. By decomposing the attention computation for long sequences into chunk-based modules, DCA manages to effectively capture the relative positional information of tokens within the same chunk (Intra-Chunk) and across distinct chunks (Inter-Chunk), as well as integrates seamlessly with Flash Attention. In…

  4. Very close Cites this paper
    Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation
    Linda He, Jue Wang, Maurice Weber et al. · 2025 · International Conference on Learning Representations · cited 8 times · Not summarized yet
    Abstract

    Large Language Models (LLMs) struggle with long-context reasoning, not only due to the quadratic scaling of computational complexity with sequence length but also because of the scarcity and expense of annotating long-context data. There has been barely any open-source work that systematically ablates long-context data, nor is there any openly available instruction tuning dataset with contexts surpassing 100K tokens. To bridge this gap, we introduce a novel post-training synthetic data generation strategy designed to efficiently extend the context window of LLMs while preserving their general task performance. Our approach scalably extends to arbitrarily long context lengths, unconstrained b…

  5. Very close
    Extending Llama-3's Context Ten-Fold Overnight
    Peitian Zhang, Ninglu Shao, Zheng Liu et al. · 2024 · Computation and Language · preprint, not peer reviewed · cited 27 times · AI summary available
    Abstract

    We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU machine. The resulted model exhibits superior performances across a broad range of evaluation tasks, such as NIHS, topic retrieval, and long-context language understanding; meanwhile, it also well preserves the original capability over short contexts. The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4 , which indicates the LLMs' inherent (yet largely underestimated) potential to extend its original context length. In fact, the context length could be ext…

  6. Very close
    Retrieval meets Long Context Large Language Models
    Peng Xu, Wei Ping, Xianchao Wu et al. · 2023 · Computation and Language · cited 159 times · AI summary available
    Abstract

    Extending the context window of large language models (LLMs) is getting popular recently, while the solution of augmenting LLMs with retrieval has existed for years. The natural questions are: i) Retrieval-augmentation versus long context window, which one is better for downstream tasks? ii) Can both methods be combined to get the best of both worlds? In this work, we answer these questions by studying both solutions using two state-of-the-art pretrained LLMs, i.e., a proprietary 43B GPT and LLaMA2-70B. Perhaps surprisingly, we find that LLM with 4K context window using simple retrieval-augmentation at generation can achieve comparable performance to finetuned LLM with 16K context window via…

  7. Very close
    BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
    Yuri Kuratov, Aydar Bulatov, Petr Anokhin et al. · 2024 · Computation and Language · cited 312 times · Not summarized yet
    Abstract

    In recent years, the input context sizes of large language models (LLMs) have increased dramatically. However, existing evaluation methods have not kept pace, failing to comprehensively assess the efficiency of models in handling long contexts. To bridge this gap, we introduce the BABILong benchmark, designed to test language models' ability to reason across facts distributed in extremely long documents. BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. These tasks are challenging on their own, and even more demanding when the required facts are scattered across long natural text. Our evaluations sh…

  8. Very close
    LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
    Yukang Chen, Shengju Qian, Haotian Tang et al. · 2023 · Computation and Language · cited 310 times · AI summary available
    Abstract

    We present LongLoRA, an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs), with limited computation cost. Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources. For example, training on the context length of 8192 needs 16x computational costs in self-attention layers as that of 2048. In this paper, we speed up the context extension of LLMs in two aspects. On the one hand, although dense global attention is needed during inference, fine-tuning the model can be effectively and efficiently done by sparse local attention. The proposed shift short attention effecti…

Prior work it builds on

  1. Very close Used in depth by this paper A method it uses
    Extending Context Window of Large Language Models via Positional Interpolation
    Shouyuan Chen, Sherman Wong, Liangjian Chen et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 924 times · AI summary available
    Abstract

    We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained conte…

  2. Very close Used in depth by this paper A method it uses
    YaRN: Efficient Context Window Extension of Large Language Models
    Bowen Peng, Jeffrey Quesnelle, Honglu Fan et al. · 2023 · Computation and Language · cited 708 times · AI summary available
    Abstract

    Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN ex…

  3. Very close A method it uses
    L-Eval: Instituting Standardized Evaluation for Long Context Language Models
    Chen An, Shansan Gong, Ming Zhong et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cited 320 times · Not summarized yet
    Abstract

    Recently, there has been growing interest in extending the context length of large language models (LLMs), aiming to effectively process long inputs of one turn or conversations with more extensive histories. While proprietary models such as GPT-4 and Claude can largely preserve the reasoning ability in an extended context, open-source models are still progressing through the early stages of development. To bridge this gap, we propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) addressing two key aspects: dataset construction and evaluation metrics. On the one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long documents, an…

  4. Close A method it uses
    PaLM: Scaling Language Modeling with Pathways
    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin et al. · 2022 · Computation and Language · cited 8,589 times · AI summary available
    Abstract

    Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning resu…

  5. Close A method it uses
    Focused Transformer: Contrastive Training for Context Scaling
    Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek et al. · 2023 · Neural Information Processing Systems · cited 191 times · Not summarized yet
    Abstract

    Large language models have an exceptional capability to incorporate new information in a contextual manner. However, the full potential of such an approach is often restrained due to a limitation in the effective context length. One solution to this issue is to endow an attention layer with access to an external memory, which comprises of (key, value) pairs. Yet, as the number of documents increases, the proportion of relevant keys to irrelevant ones decreases, leading the model to focus more on the irrelevant keys. We identify a significant challenge, dubbed the distraction issue, where keys linked to different semantic values might overlap, making them hard to distinguish. To tackle this p…

Papers that cite it

  1. Very close Uses this paper in depth
    Data Engineering for Scaling Language Models to 128K Context
    Yao Fu, Rameswar Panda, Xinyao Niu et al. · 2024 · International Conference on Machine Learning · cited 243 times · Not summarized yet
    Abstract

    We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular \textit{the ability to utilize information at arbitrary input locations}, is a capability that is mostly already acquired through large-scale pretraining, and that this capability can be readily extended to contexts substantially longer than seen during training~(e.g., 4K to 128K) through lightweight continual pretraining on appropriate data mixture. We investigate the \textit{quantity} and \textit{quality} of the data for continual pretraining: (1) for quantity, we show that 500 million to 5 billion tokens …

  2. Very close Uses this paper in depth
    ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
    Peng Xu, Wei Ping, Xianchao Wu et al. · 2024 · International Conference on Learning Representations · cited 61 times · Not summarized yet
    Abstract

    In this work, we introduce ChatQA 2, an Llama 3.0-based model with a 128K context window, designed to bridge the gap between open-source LLMs and leading proprietary models (e.g., GPT-4-Turbo-2024-04-09) in long context understanding and retrieval-augmented generation (RAG) capabilities. These two capabilities are complementary to each other and essential for LLMs to process large volumes of information that cannot fit into a single prompt. We present a detailed continued training recipe to extend the context window of Llama3-70B-base from 8K to 128K tokens, along with a three-stage instruction tuning process to enhance the model's instruction-following, RAG performance, and long-context und…

  3. Very close
    RULER: What's the Real Context Size of Your Long-Context Language Models?
    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman et al. · 2024 · Computation and Language · preprint, not peer reviewed · cited 1,603 times · AI summary available
    Abstract

    The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop…

  4. Very close Uses this paper in depth
    Shifting Long-Context LLMs Research from Input to Output
    Yuhao Wu, Yushi Bai, Zhiqing Hu et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 20 times · Not summarized yet
    Abstract

    Recent advancements in long-context Large Language Models (LLMs) have primarily concentrated on processing extended input contexts, resulting in significant strides in long-context comprehension. However, the equally critical aspect of generating long-form outputs has received comparatively less attention. This paper advocates for a paradigm shift in NLP research toward addressing the challenges of long-output generation. Tasks such as novel writing, long-term planning, and complex reasoning require models to understand extensive contexts and produce coherent, contextually rich, and logically consistent extended text. These demands highlight a critical gap in current LLM capabilities. We und…

  5. Very close
    Long-context LLMs Struggle with Long In-context Learning
    Tianle Li, Ge Zhang, Quy Duc Do et al. · 2024 · Trans. Mach. Learn. Res. · cited 415 times · Not summarized yet
    Abstract

    Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even to be capable of dealing with millions of tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities in more challenging, real-world scenarios. We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on…

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 4 November 2026.

The illustrated guide to the papers around a paper

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.