# Fast Feedforward Networks

**Authors:** Peter Belcak, Roger Wattenhofer  
**arXiv:** [2308.14711](https://arxiv.org/abs/2308.14711)  
**Date:** 2023-08-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.14711v2/

## Key points

- Fast feedforward (FFF) architecture introduced as a log-time alternative to traditional feedforward networks
- FFF architecture breaks the linear link between layer size and inference cost, leading to improvements in speed and efficiency
- FFFs outperform feedforward networks by up to 220 times in terms of speed
- FFFs are up to 6 times faster than mixture-of-experts networks commonly used for complex tasks
- FFFs exhibit superior training properties compared to mixtures of experts due to noiseless conditional execution
- FFFs can achieve remarkable efficiency in vision transformers by utilizing as little as 1% of layer neurons while preserving 94.2% predictive performance
- Potential for significantly improving inference speed and efficiency compared to traditional feedforward networks and mixture-of-experts networks
- Better training properties thanks to noiseless conditional execution

## Summary

In their paper titled "Fast Feedforward Networks," authors Peter Belcak and Roger Wattenhofer introduce the fast feedforward (FFF) architecture as a log-time alternative to traditional feedforward networks. The FFF architecture breaks the linear link between layer size and inference cost, offering significant improvements in speed and efficiency. The authors demonstrate that FFFs outperform feedforward networks by up to 220 times in terms of speed. Additionally, FFFs are up to 6 times faster than mixture-of-experts networks, which are commonly used for complex tasks. Moreover, FFFs exhibit superior training properties compared to mixtures of experts due to noiseless conditional execution. By pushing the limits of FFFs, the authors show that these networks can achieve remarkable efficiency in vision transformers. They reveal that FFFs can utilize as little as 1% of layer neurons for inference while still preserving an impressive 94.2% of predictive performance. Overall, this research presents a groundbreaking approach to neural network architecture with the introduction of the fast feedforward (FFF) architecture. The findings highlight its potential for significantly improving inference speed and efficiency compared to traditional feedforward networks and mixture-of-experts networks. Furthermore, FFFs offer better training properties thanks to noiseless conditional execution. The study's results also demonstrate how FFFs can achieve exceptional efficiency in vision transformers by utilizing a minimal percentage of layer neurons while maintaining high predictive performance levels.