Full Stack Optimization of Transformer Inference: a Survey

AI-generated keywords: Deep Neural Network

AI-generated Key Points

The license of the paper does not allow us to build upon its content and the key points are generated using the paper metadata rather than the full article.

  • Recent advances in deep neural network (DNN) architecture design are increasingly focused on Transformer models, known for superior accuracy across various applications.
  • The efficiency of recent Transformer models is a challenge due to the significant compute and bandwidth required for inference in latency-sensitive applications.
  • Various methods have been explored to enhance the efficiency of Transformer models, including modifying architecture design and developing specialized accelerators.
  • Bottlenecks in existing Transformer architectures are identified and compared with previous convolutional models, highlighting the importance of optimization strategies.
  • Techniques for optimizing Transformer models through neural architecture search are discussed, showing significant speedups in inference performance when applied to Gemmini, an open-source DNN accelerator generator.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W. Mahoney, Yakun Sophia Shao, Amir Gholami

Presented in Workshop on Architecture and System Support for Transformer Models (ASSYST) at ISCA 2023

Abstract: Recent advances in state-of-the-art DNN architecture design have been moving toward Transformer models. These models achieve superior accuracy across a wide range of applications. This trend has been consistent over the past several years since Transformer models were originally introduced. However, the amount of compute and bandwidth required for inference of recent Transformer models is growing at a significant rate, and this has made their deployment in latency-sensitive applications challenging. As such, there has been an increased focus on making Transformer models more efficient, with methods that range from changing the architecture design, all the way to developing dedicated domain-specific accelerators. In this work, we survey different approaches for efficient Transformer inference, including: (i) analysis and profiling of the bottlenecks in existing Transformer architectures and their similarities and differences with previous convolutional models; (ii) implications of Transformer architecture on hardware, including the impact of non-linear operations such as Layer Normalization, Softmax, and GELU, as well as linear operations, on hardware design; (iii) approaches for optimizing a fixed Transformer architecture; (iv) challenges in finding the right mapping and scheduling of operations for Transformer models; and (v) approaches for optimizing Transformer models by adapting the architecture using neural architecture search. Finally, we perform a case study by applying the surveyed optimizations on Gemmini, the open-source, full-stack DNN accelerator generator, and we show how each of these approaches can yield improvements, compared to previous benchmark results on Gemmini. Among other things, we find that a full-stack co-design approach with the aforementioned methods can result in up to 88.7x speedup with a minimal performance degradation for Transformer inference.

Submitted to arXiv on 27 Feb. 2023

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

The license of the paper does not allow us to build upon its content and the AI assistant only knows about the paper metadata rather than the full article.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2302.14017v1

This paper's license doesn't allow us to build upon its content and the summarizing process is here made with the paper's metadata rather than the article.

, , , , Recent advances in deep neural network (DNN) architecture design have been increasingly focused on Transformer models, which have demonstrated superior accuracy across a wide range of applications. This shift towards Transformers has been consistent over the past few years since their introduction. However, the inference of recent Transformer models requires a significant amount of compute and bandwidth, posing challenges for deployment in latency-sensitive applications. To address these challenges, there has been a growing emphasis on enhancing the efficiency of Transformer models. Various methods have been explored, ranging from modifying the architecture design to developing specialized domain-specific accelerators. In this study, different approaches for efficient Transformer inference are surveyed and analyzed. The bottlenecks in existing Transformer architectures are identified and compared with previous convolutional models. Additionally, the implications of Transformer architecture on hardware are examined, including the effects of non-linear operations such as Layer Normalization, Softmax, and GELU, as well as linear operations. Strategies for optimizing a fixed Transformer architecture are also discussed along with challenges associated with mapping and scheduling operations for Transformer models. Furthermore, techniques for optimizing Transformer models through neural architecture search are explored. A case study is presented where these optimization approaches are applied to Gemmini, an open-source DNN accelerator generator. The results demonstrate that employing a full-stack co-design approach with these methods can lead to significant speedups in Transformer inference performance. Specifically, improvements of up to 88.7x were observed with minimal degradation in performance compared to previous benchmark results on Gemmini. This research was conducted by Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh,Qijing Huang,Kurt Keutzer, Michael W. Mahoney,Yakun Sophia Shao,and Amir Gholami, and was presented at the Workshop on Architecture and System Support for Transformer Models (ASSYST) at ISCA 2023.
Created on 05 Jun. 2025

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.