# Longformer: The Long-Document Transformer

**Authors:** Iz Beltagy, Matthew E. Peters, Arman Cohan  
**arXiv:** [2004.05150](https://arxiv.org/abs/2004.05150)  
**Date:** 2020-04-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2004.05150v1/

## Key points

- The Longformer is a transformer-based model that addresses the limitation of processing long sequences due to the quadratic scaling of self-attention operation with sequence length.
- The attention mechanism in Longformer scales linearly with sequence length, making it possible to process documents of thousands of tokens or longer.
- Longformer combines a local windowed attention with a task-motivated global attention and is a drop-in replacement for standard self-attention.
- Longformer achieved state-of-the-art results on character-level language modeling on text8 and enwik8, following prior work on long-sequence transformers.
- Longformer has been pretrained and finetuned on various downstream tasks, consistently outperforming RoBERTa on long document tasks and setting new state-of-the art results on WikiHop and TriviaQA.
- Authors: Iz Beltagy, Matthew E. Peters, and Arman Cohan.
- Overall, the Longformer provides an efficient solution for processing long documents in natural language processing tasks.

## Summary

The Longformer is a new transformer-based model that addresses the limitation of processing long sequences due to the quadratic scaling of self-attention operation with sequence length. The Longformer introduces an attention mechanism that scales linearly with sequence length, making it possible to process documents of thousands of tokens or longer. This attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task-motivated global attention. The Longformer has been evaluated on character-level language modeling and achieved state-of-the-art results on text8 and enwik8, following prior work on long-sequence transformers. In contrast to most prior work, the Longformer has also been pretrained and finetuned on various downstream tasks, consistently outperforming RoBERTa on long document tasks and setting new state-of-the art results on WikiHop and TriviaQA. The authors of the paper are Iz Beltagy, Matthew E. Peters, and Arman Cohan. Overall, the Longformer provides an efficient solution for processing long documents in natural language processing tasks, enabling improved performance in downstream tasks such as WikiHop and TriviaQA compared to other models like RoBERTa.