# Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks

**Authors:** Chunjie Luo, Jianfeng Zhan, Lei Wang, Qiang Yang  
**arXiv:** [1702.05870](https://arxiv.org/abs/1702.05870)  
**Date:** 2017-02-20  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1702.05870v5/

## Key points

- Authors propose a new technique for normalizing multi-layer neural networks called cosine normalization
- Traditional dot product between output vector and incoming weight vector increases risk of large variance in neurons, resulting in poor generalization
- Cosine normalization uses cosine similarity or centered cosine similarity instead of dot product to address this issue
- Experiments show that cosine normalization outperforms other normalization techniques in terms of performance on fully-connected neural networks and convolutional networks using various datasets
- Cosine normalization helps bound dot product, decrease variance, improve generalization, and training speed by reducing internal covariate shift.
- The proposed technique has potential applications across various domains where deep learning is used.

## Summary

In their paper titled "Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks," authors Chunjie Luo, Jianfeng Zhan, Lei Wang, and Qiang Yang propose a new technique for normalizing multi-layer neural networks. Traditionally, dot product between the output vector of the previous layer and the incoming weight vector is used as input to the activation function. However, this unbounded dot product increases the risk of large variance in neurons, making models sensitive to changes in input distribution and resulting in poor generalization. To address this issue, the authors propose using cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks - a technique they call cosine normalization. The authors compare cosine normalization with batch, weight, and layer normalization techniques on fully-connected neural networks as well as convolutional networks using datasets such as MNIST, 20NEWS GROUP, CIFAR-10/100 and SVHN. Their experiments show that cosine normalization outperforms other normalization techniques in terms of performance. The proposed technique helps bound dot product and decrease variance while improving generalization and training speed by reducing internal covariate shift. Overall, this paper presents a promising new approach to normalizing multi-layer neural networks that has potential applications across various domains where deep learning is used.