# Full Stack Optimization of Transformer Inference: a Survey

**Authors:** Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W. Mahoney, Yakun Sophia Shao, Amir Gholami  
**arXiv:** [2302.14017](https://arxiv.org/abs/2302.14017)  
**Date:** 2023-02-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.14017v1/

## Key points

- Recent advances in deep neural network (DNN) architecture design are increasingly focused on Transformer models, known for superior accuracy across various applications.
- The efficiency of recent Transformer models is a challenge due to the significant compute and bandwidth required for inference in latency-sensitive applications.
- Various methods have been explored to enhance the efficiency of Transformer models, including modifying architecture design and developing specialized accelerators.
- Bottlenecks in existing Transformer architectures are identified and compared with previous convolutional models, highlighting the importance of optimization strategies.
- Techniques for optimizing Transformer models through neural architecture search are discussed, showing significant speedups in inference performance when applied to Gemmini, an open-source DNN accelerator generator.

## Summary

, , , , 

Recent advances in deep neural network (DNN) architecture design have been increasingly focused on Transformer models, which have demonstrated superior accuracy across a wide range of applications. This shift towards Transformers has been consistent over the past few years since their introduction. However, the inference of recent Transformer models requires a significant amount of compute and bandwidth, posing challenges for deployment in latency-sensitive applications. To address these challenges, there has been a growing emphasis on enhancing the efficiency of Transformer models. Various methods have been explored, ranging from modifying the architecture design to developing specialized domain-specific accelerators. In this study, different approaches for efficient Transformer inference are surveyed and analyzed. The bottlenecks in existing Transformer architectures are identified and compared with previous convolutional models. Additionally, the implications of Transformer architecture on hardware are examined, including the effects of non-linear operations such as Layer Normalization, Softmax, and GELU, as well as linear operations. Strategies for optimizing a fixed Transformer architecture are also discussed along with challenges associated with mapping and scheduling operations for Transformer models. Furthermore, techniques for optimizing Transformer models through neural architecture search are explored. A case study is presented where these optimization approaches are applied to Gemmini, an open-source DNN accelerator generator. The results demonstrate that employing a full-stack co-design approach with these methods can lead to significant speedups in Transformer inference performance. Specifically, improvements of up to 88.7x were observed with minimal degradation in performance compared to previous benchmark results on Gemmini. This research was conducted by Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh,Qijing Huang,Kurt Keutzer, Michael W. Mahoney,Yakun Sophia Shao,and Amir Gholami, and was presented at the Workshop on Architecture and System Support for Transformer Models (ASSYST) at ISCA 2023.