SIFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency

The use of weight sparsity has been explored in recent works to improve the training efficiency of deep neural networks (DNNs) by reducing training FLOPs. However, training with sparse weights often leads to accuracy loss or requires longer train schedules, making the resulting training efficiency less clear. In contrast, a new approach called SIFT (Sparse Iso-FLOP Transformations) aims to increase accuracy while using the same FLOPS as the dense model and show training efficiency gains through higher accuracy. SIFT is a family of drop-in replacements for dense layers that improve their representational capacity and FLOP efficiency. Each transformation is parameterized by a single hyperparameter (sparsity level) and provides a larger search space to find optimal sparse masks. SIFT can be used without changing any training hyperparameters and has shown significant improvements across computer vision (CV) and natural language processing (NLP) tasks, including ResNet-18 on ImageNet (+3.5%) and GPT-3 Small on WikiText-103 (-0.4 PPL). These results match larger dense model variants with 2x or more FLOPs. This is the first work to demonstrate the use of sparsity for improving accuracy of dense models via a simple-to-use set of sparse transformations. The method is explained for fully connected neural networks but can be extended straightforwardly to convolutional layers. The feedforward function fθl computes output features as a linear transformation of input features, but most transformations are expressed as dense matrix multiplications due to widespread support on GPUs. SIFT uses unstructured sparsity in weight matrices and ensures that the FLOPs of the transformation are the same as that of a dense feedforward function. In addition, detailed metrics such as AP, AP50, AP75, MIO can be found in Appendix C.2 for further evaluation. Code is available at https://github.com/CerebrasResearch/SIFT.

The Use of Weight Sparsity to Improve Training Efficiency of Deep Neural Networks

The Use of Weight Sparsity to Improve Training Efficiency of Deep Neural Networks

Introduction:

The use of weight sparsity has been explored in recent works to improve the training efficiency of deep neural networks (DNNs) by reducing training FLOPs. However, training with sparse weights often leads to accuracy loss or requires longer train schedules, making the resulting training efficiency less clear. In contrast, a new approach called SIFT (Sparse Iso-FLOP Transformations) aims to increase accuracy while using the same FLOPS as the dense model and show training efficiency gains through higher accuracy.

SIFT Overview:

SIFT is a family of drop-in replacements for dense layers that improve their representational capacity and FLOP efficiency. Each transformation is parameterized by a single hyperparameter (sparsity level) and provides a larger search space to find optimal sparse masks. SIFT can be used without changing any training hyperparameters and has shown significant improvements across computer vision (CV) and natural language processing (NLP) tasks, including ResNet-18 on ImageNet (+3.5%) and GPT-3 Small on WikiText-103 (-0.4 PPL). These results match larger dense model variants with 2x or more FLOPs.

This is the first work to demonstrate the use of sparsity for improving accuracy of dense models via a simple-to-use set of sparse transformations. The method is explained for fully connected neural networks but can be extended straightforwardly to convolutional layers.

The feedforward function fθl computes output features as a linear transformation of input features, but most transformations are expressed as dense matrix multiplications due to widespread support on GPUs. SIFT uses unstructured sparsity in weight matrices and ensures that the FLOPs of the transformation are the same as that of a dense feedforward function.

In addition, detailed metrics such as AP, AP50, AP75, MIO can be found in Appendix C.2 for further evaluation.(Code available at https://github.com/CerebrasResearch/SIFT).

Conclusion

This research paper demonstrates how weight sparsity can be used effectively in order to improve DNN's performance while reducing its computational cost significantly without compromising its accuracy levels significantly compared with denser models with more than double amount if Flops consumed by them . It also explains how this technique could be applied both for fully connected neural networks as well as convolutional layers which makes it even more versatile when it comes down into practical applications . Finally , it provides detailed metrics such us AP ,AP50 ,AP75 & MIO which helps researchers evaluate their results better .