# Simplifying Transformer Blocks

**Authors:** Bobby He, Thomas Hofmann  
**arXiv:** [2311.01906](https://arxiv.org/abs/2311.01906)  
**Date:** 2023-11-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.01906v1/

## Key points

- The standard transformer block used in deep Transformers is complex
- Complexity can lead to reduced training speed or untrainable models with minor changes
- Authors propose modifications to simplify the standard transformer block
- Modifications include removing skip connections, projection or value parameters, sequential sub-blocks, and normalization layers
- Simplified transformers achieve comparable training speed and performance to standard transformers
- Simplified transformers have a 15% faster training throughput and use 15% fewer parameters
- Simplified blocks perform well on different datasets and architectures, including bidirectional encoder-only BERT models
- Low-resource setting with 24-hour training on a single consumer GPU is used for experiments
- Results show that simplified blocks match pre-training speed of baseline within given runtime
- Proposed modifications improve training speed without sacrificing performance and reduce parameter count

## Summary

The standard transformer block used in deep Transformers is complex, with attention and MLP sub-blocks interwoven with skip connections and normalization layers. This complexity can make the architecture brittle, leading to reduced training speed or untrainable models with minor changes. In this work, the authors investigate how much the standard transformer block can be simplified. By combining signal propagation theory and empirical observations, they propose modifications that allow for the removal of several block components without sacrificing training speed. These include skip connections, projection or value parameters, sequential sub-blocks, and normalization layers. The authors conduct experiments on autoregressive decoder-only models and BERT encoder-only models. Their simplified transformers achieve comparable per-update training speed and performance to standard transformers while enjoying a 15% faster training throughput and using 15% fewer parameters. In addition to decoder-only models, the authors demonstrate that their simplified blocks also perform well on different datasets and architectures, including bidirectional encoder-only BERT models for masked language modeling tasks. They adopt a low-resource setting where the model is trained for 24 hours on a single consumer GPU. The results show that their simplified blocks match the pre-training speed of the baseline within the given runtime. Overall, this work presents a simpler design recipe for deep Transformers by removing unnecessary components from the standard transformer block. The proposed modifications improve training speed without sacrificing performance and reduce parameter count, making them more efficient than traditional transformers.