# Optimizing Neural Networks in the Equivalent Class Space

**Authors:** Qi Meng, Wei Chen, Shuxin Zheng, Qiwei Ye, Tie-Yan Liu  
**arXiv:** [1802.03713](https://arxiv.org/abs/1802.03713)  
**Date:** 2018-02-11  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1802.03713v1/

## Key points

- Authors address challenges of optimizing neural network models due to rescaling-invariant properties
- Popular activation functions and pooling methods exhibit rescaling-invariant properties
- Rescaling invariance can lead to issues during optimization
- Two main problems highlighted: functionally equivalent models with different gradient behaviors, loss functions containing spurious critical points in redundant weight space
- Proposed approach involves characterizing rescaling-invariant properties using equivalent classes
- Developed Equivalent Class Optimization (EC-Opt) algorithms to streamline optimization process within reduced-dimensional space
- Introduced efficient techniques for computing gradients in equivalent class with minimal additional computational complexity
- Experimental study demonstrates effectiveness of EC-Opt algorithms in enhancing model accuracy compared to traditional approaches

## Summary

In their paper "Optimizing Neural Networks in the Equivalent Class Space," authors Qi Meng, Wei Chen, Shuxin Zheng, Qiwei Ye, and Tie-Yan Liu address the challenges of optimizing neural network models due to their rescaling-invariant properties. These properties are exhibited by popular activation functions like ReLU and PReLU as well as pooling methods such as max-pooling and average pooling. While these characteristics make fully-connected neural networks (FNNs) and convolutional neural networks (CNNs) insensitive to positive rescaling operations across layers, they can also lead to significant issues during optimization. The authors highlight two main problems arising from the rescaling-invariant nature of neural networks: different models may be functionally equivalent but exhibit vastly different gradient behaviors, and the associated loss functions may contain numerous spurious critical points within the redundant weight space. To tackle these challenges, they propose a novel approach that involves characterizing the rescaling-invariant properties of neural network models using equivalent classes. By representing the loss function in a compact equivalent class space and developing innovative optimization algorithms known as Equivalent Class Optimization (EC-Opt) algorithms, the researchers aim to streamline the optimization process directly within this reduced-dimensional space. They also introduce efficient techniques for computing gradients in the equivalent class with minimal additional computational complexity compared to standard back-propagation methods. Through an experimental study outlined in their paper, Meng et al. demonstrate the effectiveness of their proposed EC-Opt algorithms in enhancing model accuracy for both fully-connected and convolutional neural networks when compared to traditional stochastic gradient descent approaches. By leveraging the concept of equivalent classes and optimizing within this specialized space, significant improvements in model performance are achieved - showcasing a promising avenue for advancing neural network optimization strategies.