# Hyena Hierarchy: Towards Larger Convolutional Language Models

**Authors:** Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, Christopher Ré  
**arXiv:** [2302.10866](https://arxiv.org/abs/2302.10866)  
**Date:** 2023-02-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.10866v1/

## Key points

- Deep learning has made significant strides in NLP tasks, with large Transformers being a popular choice.
- Attention operator - a core building block of Transformers - exhibits quadratic cost in sequence length, which limits the amount of context that can be accessed.
- Existing subquadratic methods based on low-rank and sparse approximations have been developed but still need to be combined with dense attention layers to match the performance of Transformers.
- Michael Poli and his team propose Hyena - a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating.
- In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models.
- The team also set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K.
- Hyena operators are twice as fast as highly optimized attention at sequence length 8K and 100x faster at sequence length 64K.
- Overall, Hyena offers an efficient alternative to traditional attention mechanisms used in NLP tasks. Its success highlights the potential for further research into developing more efficient deep learning models that can handle larger sequences while maintaining high levels of accuracy.

## Summary

In recent years, deep learning has made significant strides in natural language processing (NLP) tasks, with large Transformers being a popular choice due to their ability to learn at scale. However, the attention operator - a core building block of Transformers - exhibits quadratic cost in sequence length, which limits the amount of context that can be accessed. While existing subquadratic methods based on low-rank and sparse approximations have been developed, they still need to be combined with dense attention layers to match the performance of Transformers. This indicates a gap in capability that needs to be addressed. To address this issue, Michael Poli and his team propose Hyena - a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. The team also set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Furthermore, Hyena operators are twice as fast as highly optimized attention at sequence length 8K and 100x faster at sequence length 64K. Overall, Hyena offers an efficient alternative to traditional attention mechanisms used in NLP tasks. Its success highlights the potential for further research into developing more efficient deep learning models that can handle larger sequences while maintaining high levels of accuracy.