# Linearizing Transformer with Key-Value Memory Bank

**Authors:** Yizhe Zhang, Deng Cai  
**arXiv:** [2203.12644](https://arxiv.org/abs/2203.12644)  
**Date:** 2022-03-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.12644v1/

## Key points

- Paper introduces MemSizer, a new approach for addressing computational overhead of vanilla transformer in NLP tasks
- Vanilla transformer's complexity scales quadratically with sequence length
- Previous work like Linformer achieves linear time complexity but not suitable for text generation tasks
- MemSizer proposes different perspective on attention mechanism and projects source sequence into lower dimension representation
- MemSizer can handle input sequences with dynamic lengths, making it more suitable for text generation tasks
- MemSizer achieves linear time complexity and offers efficient recurrent-style autoregressive generation
- Constant memory complexity and reduced computation during inference
- MemSizer strikes improved balance between efficiency and accuracy compared to vanilla transformer and other linear variants in language modeling and machine translation tasks
- MemSizer presented as efficient alternative to vanilla transformer by leveraging key-value memory banks and offering dynamic length support for text generation tasks
- Experimental results showcase effectiveness of MemSizer in achieving better tradeoffs between efficiency and accuracy compared to existing approaches.

## Summary

, , , , 

The paper titled "Linearizing Transformer with Key-Value Memory Bank" by Yizhe Zhang and Deng Cai introduces MemSizer, a new approach for addressing the computational overhead of the vanilla transformer in natural language processing tasks. The vanilla transformer has achieved great success, but its complexity scales quadratically with sequence length. Previous work such as Linformer has attempted to overcome this limitation by projecting the input sequence into a low-rank space, achieving linear time complexity. However, Linformer is not suitable for text generation tasks as it requires pre-specification of the sequence length. In contrast, MemSizer proposes a different perspective on the attention mechanism and projects the source sequence into a lower dimension representation. What sets MemSizer apart is its ability to handle input sequences with dynamic lengths, making it more suitable for text generation tasks. Similar to Linformer, MemSizer achieves linear time complexity but also offers efficient recurrent-style autoregressive generation. This results in constant memory complexity and reduced computation during inference. The authors demonstrate that MemSizer strikes an improved balance between efficiency and accuracy compared to both the vanilla transformer and other linear variants in language modeling and machine translation tasks. This highlights MemSizer as a viable direction for further improving inference efficiency in natural language processing. Overall, this paper presents MemSizer as an efficient alternative to the vanilla transformer by leveraging key-value memory banks and offering dynamic length support for text generation tasks. The experimental results showcase its effectiveness in achieving better tradeoffs between efficiency and accuracy compared to existing approaches.