# Switch EMA: A Free Lunch for Better Flatness and Sharpness

**Authors:** Siyuan Li, Zicheng Liu, Juanxi Tian, Ge Wang, Zedong Wang, Weiyang Jin, Di Wu, Cheng Tan, Tao Lin, Yang Liu, Baigui Sun, Stan Z. Li  
**arXiv:** [2402.09240](https://arxiv.org/abs/2402.09240)  
**Date:** 2024-02-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.09240v2/

## Key points

- Switch EMA (SEMA) enhances DNN performance by leveraging Exponential Moving Average (EMA) in weight averaging regularization
- SEMA involves switching EMA parameters back to the original model after each epoch, leading to significant improvements without extra computational cost
- Integration of SEMA into training process helps DNNs achieve optimal generalization with a balance between flatness and sharpness
- SEMA outperforms existing methods across various tasks like image classification, self-supervised learning, object detection, image generation, video prediction, attribute regression, and language modeling
- Research by Siyuan Li et al. shows that SEMA is a "free lunch" for DNN training, improving final performances and convergence speeds across different optimizers and network architectures
- Source code and models for further exploration available on GitHub (https://github.com/Westlake-AI/SEMA), making SEMA a game-changing technique for better flatness and sharpness in deep learning optimization

## Summary

Switch EMA (SEMA) is a groundbreaking approach that enhances the performance of deep neural networks (DNNs) by leveraging the power of Exponential Moving Average (EMA) in weight averaging regularization. This innovative technique involves a simple modification - switching the EMA parameters back to the original model after each epoch - resulting in significant improvements without any additional computational cost. By seamlessly integrating SEMA into the training process, DNNs are able to achieve optimal generalization with a balanced trade-off between flatness and sharpness. The effectiveness of SEMA is validated through comprehensive experiments across various tasks including discriminative, generative, and regression tasks on both vision and language datasets. From image classification to self-supervised learning, object detection and segmentation, image generation, video prediction, attribute regression, and language modeling, SEMA consistently outperforms existing methods by enhancing convergence speeds and overall performance. The research conducted by Siyuan Li et al. demonstrates that SEMA serves as a "free lunch" for DNN training. The findings highlight how this novel approach not only improves final performances but also boosts convergence speeds across different optimizers and network architectures. With source code and models available for further exploration on GitHub (https://github.com/Westlake-AI/SEMA), SEMA emerges as a game-changing technique that unlocks the full potential of EMA for better flatness and sharpness in deep learning optimization.