# SIFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency

**Authors:** Shreyas Saxena, Vithursan Thangarasa, Abhay Gupta, Sean Lie  
**arXiv:** [2303.11525](https://arxiv.org/abs/2303.11525)  
**Date:** 2023-03-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.11525v1/

## Key points

- Weight sparsity has been explored to improve training efficiency of deep neural networks (DNNs) by reducing training FLOPs.
- Sparse weights often lead to accuracy loss or require longer train schedules, making the resulting training efficiency less clear.
- SIFT (Sparse Iso-FLOP Transformations) is a new approach that aims to increase accuracy while using the same FLOPS as the dense model and show training efficiency gains through higher accuracy.
- SIFT is a family of drop-in replacements for dense layers that improve their representational capacity and FLOP efficiency.
- Each transformation is parameterized by a single hyperparameter (sparsity level) and provides a larger search space to find optimal sparse masks.
- SIFT can be used without changing any training hyperparameters and has shown significant improvements across computer vision (CV) and natural language processing (NLP) tasks, including ResNet-18 on ImageNet (+3.5%) and GPT-3 Small on WikiText-103 (-0.4 PPL).
- The method is explained for fully connected neural networks but can be extended straightforwardly to convolutional layers.
- SIFT uses unstructured sparsity in weight matrices and ensures that the FLOPs of the transformation are the same as that of a dense feedforward function.
- Detailed metrics such as AP, AP50, AP75, MIO can be found in Appendix C.2 for further evaluation.
- Code is available at https://github.com/CerebrasResearch/SIFT.

## Summary

The use of weight sparsity has been explored in recent works to improve the training efficiency of deep neural networks (DNNs) by reducing training FLOPs. However, training with sparse weights often leads to accuracy loss or requires longer train schedules, making the resulting training efficiency less clear. In contrast, a new approach called SIFT (Sparse Iso-FLOP Transformations) aims to increase accuracy while using the same FLOPS as the dense model and show training efficiency gains through higher accuracy. SIFT is a family of drop-in replacements for dense layers that improve their representational capacity and FLOP efficiency. Each transformation is parameterized by a single hyperparameter (sparsity level) and provides a larger search space to find optimal sparse masks. SIFT can be used without changing any training hyperparameters and has shown significant improvements across computer vision (CV) and natural language processing (NLP) tasks, including ResNet-18 on ImageNet (+3.5%) and GPT-3 Small on WikiText-103 (-0.4 PPL). These results match larger dense model variants with 2x or more FLOPs. This is the first work to demonstrate the use of sparsity for improving accuracy of dense models via a simple-to-use set of sparse transformations. The method is explained for fully connected neural networks but can be extended straightforwardly to convolutional layers. The feedforward function fθl computes output features as a linear transformation of input features, but most transformations are expressed as dense matrix multiplications due to widespread support on GPUs. SIFT uses unstructured sparsity in weight matrices and ensures that the FLOPs of the transformation are the same as that of a dense feedforward function. In addition, detailed metrics such as AP, AP50, AP75, MIO can be found in Appendix C.2 for further evaluation. Code is available at https://github.com/CerebrasResearch/SIFT.