# Maximum Class Separation as Inductive Bias in One Matrix

**Authors:** Tejaswi Kasarla, Gertjan J. Burghouts, Max van Spengler, Elise van der Pol, Rita Cucchiara, Pascal Mettes  
**arXiv:** [2206.08704](https://arxiv.org/abs/2206.08704)  
**Date:** 2022-06-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.08704v1/

## Key points

- The paper proposes a solution to maximize separation between classes in machine learning.
- Maximum separation is a well-known inductive bias used in traditional algorithms but not equipped by default in deep networks.
- The authors suggest encoding maximum separation as an inductive bias by adding one fixed matrix multiplication before computing the softmax activations.
- Separation does not require optimization but can be solved in closed-form prior to training and plugged into a network.
- A recursive approach is outlined to obtain the matrix consisting of maximally separable vectors for any number of classes with negligible engineering effort and computational overhead.
- This one matrix multiplication provides real impact on classification, long-tailed recognition, out-of-distribution detection, and open-set recognition from CIFAR to ImageNet.
- Maximum separation works best as a fixed bias; making the matrix learnable adds nothing to performance.
- Closed-form implementation and code are provided on GitHub to reproduce the experiments.
- The paper offers a straightforward solution without requiring complex optimization methods or additional computational resources.
- Traditional techniques can be adapted for use with deep neural networks, highlighting the importance of incorporating established knowledge into new technologies.

## Summary

The paper "Maximum Class Separation as Inductive Bias in One Matrix" proposes a simple and effective solution to the problem of maximizing separation between classes in machine learning. This is a well-known inductive bias that has been used in traditional algorithms but deep networks are not equipped with this bias by default. The authors suggest encoding maximum separation as an inductive bias by adding one fixed matrix multiplication before computing the softmax activations. The main observation behind their approach is that separation does not require optimization but can be solved in closed-form prior to training and plugged into a network. To obtain the matrix consisting of maximally separable vectors for any number of classes, the authors outline a recursive approach that can be added with negligible engineering effort and computational overhead. Despite its simplicity, this one matrix multiplication provides real impact on classification, long-tailed recognition, out-of-distribution detection, and open-set recognition from CIFAR to ImageNet. The authors show empirically that maximum separation works best as a fixed bias; making the matrix learnable adds nothing to performance. They provide closed-form implementation and code to reproduce the experiments on GitHub. This paper's contribution is significant because it offers a straightforward solution to an important problem in machine learning without requiring complex optimization methods or additional computational resources. It also demonstrates how traditional techniques can be adapted for use with deep neural networks, highlighting the importance of incorporating established knowledge into new technologies.