# Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon

**Authors:** Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, Zhicheng Dou  
**arXiv:** [2401.03462](https://arxiv.org/abs/2401.03462)  
**Date:** 2024-01-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.03462v1/

## Key points

- Activation Beacon is a method proposed to address the challenge of utilizing long contexts in large language models (LLMs) with limited context window length.
- Activation Beacon is introduced as a plug-and-play module for LLMs that condenses raw activations into more compact forms, allowing them to perceive longer contexts within the limited window.
- The module fully preserves the LLM's original capability on short contexts while extending its capability on processing longer contexts.
- Activation Beacon achieves competitive memory and time efficiency in both training and inference by working with short sliding windows to process the long context.
- The module is learned through an auto-regression task conditioned on a mixture of beacons with diversified condensing ratios and can be efficiently trained purely with short-sequence data in just 10K steps, consuming less than 9 hours on a single GPU machine.
- Experimental studies show that Activation Beacon extends Llama-2-7B's context length by 100 times (from 4K to 400K) and achieves superior results on both long-context generation and understanding tasks compared to fine-tuned full-attention baselines.
- Activation Beacon is further evaluated on five real-world tasks from LongBench, including single-doc QA, multi-doc QA, summarization, few-shot learning, and code completion. It achieves similar performance as fine-tuned methods like LongChat-32K and LongAlpaca-16K.
- Experiments on long context language modeling using three datasets (PG19, Proof-Pile, CodeParrot) demonstrate that Activation Beacon leads to superior long context language modeling performance compared to baseline methods and fine-tuning free methods.
- Overall, Activation Beacon proves to be an effective solution for extending the context length of LLMs without sacrificing their original capabilities. The model and code are available at the BGE repository.

## Summary

The authors propose a method called Activation Beacon to address the challenge of utilizing long contexts in large language models (LLMs) with limited context window length. <br>
Activation Beacon is introduced as a plug-and-play module for LLMs that condenses raw activations into more compact forms, allowing them to perceive longer contexts within the limited window. This module fully preserves the LLM's original capability on short contexts while extending its capability on processing longer contexts. It achieves competitive memory and time efficiency in both training and inference by working with short sliding windows to process the long context.<br>
 Activation Beacon is learned through an auto-regression task conditioned on a mixture of beacons with diversified condensing ratios. The module can be efficiently trained purely with short-sequence data in just 10K steps, consuming less than 9 hours on a single GPU machine.<br>
Experimental studies show that Activation Beacon is able to extend Llama-2-7B's context length by 100 times (from 4K to 400K), while achieving superior results on both long-context generation and understanding tasks. The method performs on par with fine-tuned full-attention baselines.<br>
The authors further evaluate Activation Beacon on five real-world tasks from LongBench, including single-doc QA, multi-doc QA, summarization, few-shot learning, and code completion. The results are reported in Table 3, showing that Activation Beacon achieves similar performance as fine-tuned methods like LongChat-32K and LongAlpaca-16K.<br>
Additionally, the authors conduct experiments on long context language modeling using three datasets: PG19, Proof-Pile, and CodeParrot. The perplexity results are reported in Table 2, demonstrating that Activation Beacon leads to superior long context language modeling performance compared to baseline methods and fine-tuning free methods.<br>
Overall, Activation Beacon proves to be an effective solution for extending the context length of LLMs without sacrificing their original capabilities. The model and code are available at the BGE repository.