# Stronger Normalization-Free Transformers

**Authors:** Mingzhi Chen, Taiming Lu, Jiachen Zhu, Mingjie Sun, Zhuang Liu  
**arXiv:** [2512.10938](https://arxiv.org/abs/2512.10938)  
**Date:** 2025-12-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2512.10938v1/

## Key points

- Normalization layers have been considered essential in deep learning architectures.
- Dynamic Tanh (DyT) is a point-wise function that ensures stable convergence and achieves performance levels comparable to traditional normalization techniques.
- Derf(x) = erf(αx + s) is introduced as a promising candidate with superior performance compared to established normalization methods like LayerNorm, RMSNorm, and DyT.
- Derf's enhanced generalization capabilities are highlighted as the primary reason for its superior performance.
- Derf is praised for its simplicity and remarkable performance gains, making it a practical choice for normalization-free Transformer architectures.
- The collaborative efforts of Mingzhi Chen, Taiming Lu, Jiachen Zhu, Mingjie Sun, and Zhuang Liu have advanced the field by introducing Stronger Normalization-Free Transformers.

## Summary

, , , , 

In the realm of deep learning architectures, normalization layers have long been considered essential components. However, a recent breakthrough in the form of Dynamic Tanh (DyT) has challenged this notion by showcasing alternative possibilities. DyT is a point-wise function that effectively constrains extreme values to ensure stable convergence and achieves performance levels comparable to traditional normalization techniques. This study delves deeper into exploring novel function designs that can outperform DyT by investigating how the intrinsic properties of point-wise functions impact training and performance. Through an extensive exploration, the team introduces Derf(x) = erf(αx + s) as a promising candidate, where erf(x) represents the rescaled Gaussian cumulative distribution function. Remarkably, Derf surpasses established normalization methods such as LayerNorm, RMSNorm, and DyT across various domains including image recognition and generation, speech representation, and DNA sequence modeling. The findings suggest that Derf's superior performance is primarily attributed to its enhanced generalization capabilities rather than just stronger fitting capacity. Notably, Derf stands out for its simplicity and remarkable performance gains compared to existing normalization techniques. As a result, it emerges as a practical choice for normalization-free Transformer architectures. The collaborative efforts of Mingzhi Chen, Taiming Lu, Jiachen Zhu, Mingjie Sun, and Zhuang Liu have significantly advanced the field by introducing Stronger Normalization-Free Transformers that pave the way for more efficient deep learning models.