# Revisiting Transformer-based Models for Long Document Classification

**Authors:** Xiang Dai, Ilias Chalkidis, Sune Darkner, Desmond Elliott  
**arXiv:** [2204.06683](https://arxiv.org/abs/2204.06683)  
**Date:** 2022-04-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2204.06683v2/

## Key points

- Bias towards short text sequences in recent literature on text classification
- Challenges with encoding multi-page multi-paragraph documents efficiently using traditional Transformer-based models
- Various approaches developed to address the issue, such as sparse attention and hierarchical encoding methods
- Strategies for splitting documents into segments for efficient encoding by pre-trained models
- Computational overhead of vanilla Transformers in long document classification due to O(n^2) time and memory complexity
- Emergence of long-document Transformers designed to handle longer sequences effectively
- Performance comparison of traditional BERT variants with CNN or RNN-based models on datasets like MIMIC-III with longer documents
- Success of pre-train–fine-tune paradigm in long document classification
- Significant contributions made by analyzing different components of Transformer-based long document classification models
- Improvements in performance demonstrated when models can process more text, especially with a sparse attention model capable of processing up to 4096 tokens
- Refutation of criticisms against Transformers for long document classification through experiments
- Valuable insights provided for effective application of Transformer-based models in handling longer texts

## Summary

In recent literature on text classification, there is a bias towards short text sequences like sentences or paragraphs. However, in real-world applications, multi-page multi-paragraph documents are common and cannot be efficiently encoded by traditional Transformer-based models. To address this issue, various  approaches have been developed, such as sparse attention and hierarchical encoding methods. involves adjusting aspects like the size of local attention windows and the use of global attention to improve efficiency. On the other hand,  focus on strategies for splitting documents into segments that can be encoded by pre-trained models. These approaches have been tested on four document classification datasets across different domains. One key challenge in long document classification is the computational overhead of vanilla Transformers due to their O(n^2) time and memory complexity in attending to all tokens in a sequence. This limitation has led to the emergence of long-document Transformers designed to handle longer sequences more effectively. Experiments conducted on datasets with longer documents, such as MIMIC-III with an average length of 2,000 words, have shown that traditional BERT variants perform worse than CNN or RNN-based models. This highlights the need to understand how Transformer-based models perform when classifying actually long documents. The study aims to transfer the success of pre-train–fine-tune paradigm to long document classification and makes significant contributions by analyzing different components of Transformer-based long document classification models. The experiments conducted on datasets like MIMIC-III demonstrate clear improvements in performance when models can process more text. , a sparse attention model capable of processing up to 4096 tokens, shows competitive results with CNN-based models on MIMIC-III. outperform CNN-based models significantly by splitting documents into small overlapping segments for efficient encoding by pre-trained models. Overall, these experiments refute criticisms against Transformers for long document classification and provide valuable insights for applying Transformer-based models effectively in handling longer texts. The study is funded by various organizations including Innovation Fund Denmark and CSIRO Precision Health Future Science Platform under projects like AI4Xray.