# Compute Better Spent: Replacing Dense Layers with Structured Matrices

**Authors:** Shikai Qiu, Andres Potapczynski, Marc Finzi, Micah Goldblum, Andrew Gordon Wilson  
**arXiv:** [2406.06248](https://arxiv.org/abs/2406.06248)  
**Date:** 2024-06-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.06248v1/

## Key points

- Authors address computational bottleneck posed by dense linear layers in foundation models
- Exploration of structured matrices as alternatives to dense matrices for more efficient model building
- Investigation of different matrix structures and their impact on initialization scales and learning rates
- Determination of optimal scaling strategies based on insights from Maximal Update Parameterization
- Analysis of scaling laws to assess performance improvement with increased computational resources
- Introduction of Monarch matrices, specifically showcasing Block Tensor-Train (BTT) within this family
- BTT demonstrates exponentially lower training loss compared to dense matrices for tasks like CIFAR-10/100 and ImageNet-1k
- BTT achieves equivalent performance to dense ViT-S/32 on ImageNet-1k with significantly less compute power
- More efficient than dense matrices for training small GPT-2 language models
- Research highlights potential of structured matrices in enhancing model efficiency and performance across various tasks

## Summary

In the study "Compute Better Spent: Replacing Dense Layers with Structured Matrices," authors Shikai Qiu, Andres Potapczynski, Marc Finzi, Micah Goldblum, and Andrew Gordon Wilson address the computational bottleneck posed by dense linear layers in foundation models. They highlight the potential for more efficient model building by exploring structured matrices as alternatives to dense matrices. Drawing inspiration from the success of convolutional networks in the image domain, the researchers systematically investigate various structured matrix configurations. Their research reveals that different matrix structures necessitate distinct initialization scales and learning rates, which significantly impact performance as models scale up. Leveraging insights from the Maximal Update Parameterization, they determine optimal scaling strategies for initialization and learning rates of these unconventional layers. Additionally, they analyze the scaling laws of different matrix structures to assess how quickly performance improves with increased computational resources. The team introduces a novel matrix family known as Monarch matrices, specifically showcasing the Block Tensor-Train (BTT) within this family. Through experiments on tasks such as CIFAR-10/100 with augmentation, BTT demonstrates exponentially lower training loss compared to dense matrices when training Multilayer Perceptrons (MLPs) and Vision Transformers (ViTs). Remarkably, BTT achieves performance equivalent to dense ViT-S/32 on ImageNet-1k while utilizing 3.8 times less compute power. Furthermore, BTT proves more efficient than dense matrices for training small GPT-2 language models. Overall, this research underscores the potential of structured matrices in enhancing model efficiency and performance across various tasks, offering valuable insights into optimizing deep learning architectures for improved computational efficiency.