# Smooth Loss Functions for Deep Top-k Classification

**Authors:** Leonard Berrada, Andrew Zisserman, M. Pawan Kumar  
**arXiv:** [1802.07595](https://arxiv.org/abs/1802.07595)  
**Date:** 2018-02-21  
**Summary page:** https://www.summarizepaper.com/en/paper/1802.07595v1/

## Key points

- Top-k error is a widely used measure of performance in machine learning and computer vision
- Cross-entropy loss is typically used to train deep neural networks for top-k classification
- Limited and noisy data can significantly impact the effectiveness of cross-entropy loss
- A loss function specifically designed for top-k classification can bring significant improvements, and must be smooth with non-sparse gradients to work well with deep neural networks
- A family of smoothed loss functions suited to top-k optimization via deep learning has been introduced, which includes cross-entropy as a special case but provides improved performance over traditional methods in scenarios with limited and noisy data conditions
- An algorithm with time complexity $\mathcal{O}(kn)$ has been utilized to efficiently evaluate these smooth loss functions, along with a novel approximation for fast and stable algorithms on GPUs with single floating point precision
- The proposed loss function is more robust to noise and overfitting than cross entropy, as shown through comparisons in various regimes of noise and data size for the predominant use case of k=5.
- This research provides insights into designing effective loss functions for top k classification tasks in deep learning under limited and noisy data conditions.

## Summary

In the field of machine learning and computer vision, the top-k error is a widely used measure of performance. Typically, deep neural networks trained with cross-entropy loss are used for top-k classification. While theoretical results suggest that cross-entropy is an optimal learning objective for this task in the limit of infinite data, limited and noisy data can significantly impact its effectiveness. To address this issue, a loss function specifically designed for top-k classification can bring significant improvements. Empirical evidence suggests that such a loss function must be smooth and have non-sparse gradients to work well with deep neural networks. To this end, a family of smoothed loss functions suited to top-k optimization via deep learning has been introduced. This family includes cross-entropy as a special case but also provides improved performance over traditional methods in scenarios with limited and noisy data conditions. To evaluate these smooth loss functions efficiently, a connection to polynomial algebra and divide-and-conquer approach has been utilized to provide an algorithm with time complexity $\mathcal{O}(kn)$. Additionally, a novel approximation has been presented to obtain fast and stable algorithms on GPUs with single floating point precision. The performance of cross entropy loss and margin based losses have been compared in various regimes of noise and data size for the predominant use case of k=5. The investigation reveals that the proposed loss function is more robust to noise and overfitting than cross entropy. Overall, this research provides insights into designing effective loss functions for top k classification tasks in deep learning under limited and noisy data conditions.