# FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

**Authors:** Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré  
**arXiv:** [2205.14135](https://arxiv.org/abs/2205.14135)  
**Date:** 2022-05-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.14135v2/

## Key points

- Authors address the challenge of slow and memory-intensive Transformers on long sequences due to quadratic time and memory complexity of self-attention
- Introduce FlashAttention, an IO-aware exact attention algorithm that minimizes memory reads/writes between GPU HBM and GPU on-chip SRAM through tiling
- FlashAttention demonstrates superiority over standard attention mechanisms in terms of HBM accesses and optimal performance for a range of SRAM sizes
- Extension to block-sparse attention results in an approximate attention algorithm that surpasses existing methods in speed
- Experimental evaluations show FlashAttention enables faster training of Transformers with significant speedups on BERT-large, GPT-2, and long-range arena sequences
- Facilitates longer context modeling leading to higher quality models with improvements in perplexity for GPT-2 and long-document classification tasks
- Achievements include outperforming chance performance on challenging tasks like Path-X and Path-256, showcasing groundbreaking advancements in natural language processing tasks

## Summary

, , , , 

In their paper titled "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," authors Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré address the challenge of slow and memory-intensive Transformers on long sequences due to the quadratic time and memory complexity of self-attention. The authors propose a novel approach by introducing FlashAttention, an IO-aware exact attention algorithm that leverages tiling to minimize the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM. By analyzing the IO complexity of FlashAttention, they demonstrate its superiority over standard attention mechanisms in terms of HBM accesses and optimal performance for a range of SRAM sizes. Additionally, they extend FlashAttention to block-sparse attention, resulting in an approximate attention algorithm that surpasses existing methods in terms of speed. Through experimental evaluations, the authors show that FlashAttention enables faster training of Transformers compared to current baselines. They achieve a 15% end-to-end wall-clock speedup on BERT-large with a sequence length of 512, a 3x speedup on GPT-2 with a sequence length of 1K, and a 2.4x speedup on long-range arena sequences ranging from 1K to 4K. Notably, FlashAttention facilitates longer context modeling in Transformers leading to higher quality models as evidenced by a 0.7 improvement in perplexity for GPT-2 and a substantial lift of 6.4 points in long-document classification tasks. Moreover, the enhanced capabilities enabled by FlashAttention are demonstrated through groundbreaking achievements such as being the first Transformers models to outperform chance performance on challenging tasks like Path-X (achieving 61.4% accuracy on sequences of length 16K) and Path-256 (achieving 63.1% accuracy on sequences of length 64K). Overall, FlashAttention represents a significant advancement in optimizing attention mechanisms for efficient Transformer training while enhancing model quality and enabling new possibilities in natural language processing tasks.