# Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens

**Authors:** Zhanpeng Zeng, Cole Hawkins, Mingyi Hong, Aston Zhang, Nikolaos Pappas, Vikas Singh, Shuai Zheng  
**arXiv:** [2305.04241](https://arxiv.org/abs/2305.04241)  
**Date:** 2023-05-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.04241v1/

## Key points

- Transformer models are foundational in NLP and computer vision
- The authors propose a VIP-token centric compression (Vcc) scheme that reduces the dependency of a Transformer model's complexity on sequence length
- Vcc selectively compresses input sequences based on their impact on approximating the representation of VIP-tokens, which are small subsets of special tokens most relevant to final predictions in many tasks
- The proposed algorithm achieves more than 3x efficiency improvement compared to baselines on 4K and 16K lengths while also achieving competitive or better performance on various tasks
- It can be scaled up to 128K tokens while consistently offering accuracy improvement
- NarrativeQA is an ideal testbed for scaling experiments since it involves longer sequences and shows that their method can be scaled to much longer sequences and achieve higher performance as sequence length increases.
- Using a few layers of standard Transformer layers to preprocess tokens helps performance and segmenting input sequence into multiple segments of 512 length before using vanilla computation in initial stages.
- For encoder-only architecture, they compare their method with RoBERTa and two strong baselines: Longformer and Big Bird.
- The proposed VIP-token centric compression scheme offers significant efficiency improvements while maintaining or improving performance on various tasks.

## Summary

In the field of natural language processing (NLP) and computer vision, Transformer models are foundational. The authors propose a VIP-token centric compression (Vcc) scheme that significantly reduces the dependency of a Transformer model's complexity on sequence length by compressing the input into a representation whose size is independent of n at each layer. The Vcc scheme selectively compresses the input sequence based on their impact on approximating the representation of VIP-tokens, which are small subsets of special tokens that are most relevant to final predictions in many tasks. Compared to competitive baselines, the proposed algorithm achieves more than 3x efficiency improvement compared to baselines on 4K and 16K lengths while also achieving competitive or better performance on a large number of tasks. Additionally, it can be scaled up to 128K tokens while consistently offering accuracy improvement. The authors note that NarrativeQA is an ideal testbed for scaling experiments since it involves longer sequences and show that their method can be scaled to much longer sequences and achieve higher performance as sequence length increases. They allow for larger ratios between parent node and child nodes in practice to reduce tree depth and restrict J = {bsx : s ∈ {1, s0}} for a pre-defined s0 to have exactly two resolutions for their experiments. They found that using a few layers of standard Transformer layers to preprocess tokens helps performance and segment input sequence into multiple segments of 512 length before using vanilla computation in initial stages. For encoder-only architecture, they compare their method with RoBERTa and two strong baselines: Longformer and Big Bird. They first pretrain a standard RoBERTa model using masked language modeling task, then do continuous pretraining from the pretrained RoBERTa checkpoint to expand positional embeddings to 4K length and adjust model parameters. Overall, the proposed VIP-token centric compression scheme offers significant efficiency improvements while maintaining or improving performance on various tasks. The authors believe that their method can be applied to shorter sequences but note that compressing irrelevant information may not offer meaningful speed up when there is less compressible information for VIP-tokens.