# A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning

**Authors:** Alicia Curth, Alan Jeffares, Mihaela van der Schaar  
**arXiv:** [2310.18988](https://arxiv.org/abs/2310.18988)  
**Date:** 2023-10-29  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2310.18988v1/

## Key points

- Relationship between model complexity and prediction error in statistical learning
- U-shaped curve depicting transition between underfitting and overfitting
- Emergence of double descent concept challenging conventional wisdom
- Test error decreasing as number of parameters surpasses sample size
- Initially observed in deep learning models, now found in classical statistical machine learning methods
- Multiple underlying complexity axes along which parameter count grows
- Second descent in test error occurs at transitions between these axes
- Location not linked to specific thresholds like interpolation threshold p=n
- Proposed measure for effective number of parameters on unseen examples
- Reconciling apparent double descent curves with traditional convex shapes in classical statistical machine learning methods

## Summary

In the realm of statistical learning, there is a well-established relationship between model complexity and prediction error. This is typically depicted as a U-shaped curve that signifies the transition between underfitting and overfitting. However, recent advancements in the field have challenged this conventional wisdom, particularly with the success of overparametrized neural networks. This has led to the emergence of a new concept known as double descent, which suggests an additional regime where test error decreases again as the number of parameters surpasses the sample size. This phenomenon defies traditional understanding and has initially been observed in deep learning models but has since been found in classical statistical machine learning methods such as linear regression, trees, and boosting. To address this paradigm shift, a closer examination of these traditional methods was undertaken to assess whether cases of double descent truly expand beyond the boundaries of the familiar U-shaped complexity-generalization curve. Upon careful analysis of the x-axes in double descent plots, it became apparent that there are multiple underlying complexity axes along which parameter count grows. The emergence of a second descent in test error occurs precisely at points where transitions between these axes take place, indicating that its location is not inherently linked to specific thresholds like interpolation threshold p=n. By adopting a classical nonparametric statistics perspective and viewing the investigated methods as smoothers rather than purely predictive models, a generalized measure for their effective number of parameters on unseen examples was proposed. Through this refined approach, it was demonstrated that the apparent double descent curves observed in classical statistical machine learning methods can be reconciled with more traditional convex shapes. This resolution provides valuable insights into understanding how model complexity impacts generalization performance and offers a new perspective on navigating complexities in statistical intuition within modern machine learning frameworks.