# Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

**Authors:** Sebastian Raschka  
**arXiv:** [1811.12808](https://arxiv.org/abs/1811.12808)  
**Date:** 2018-11-13  
**Summary page:** https://www.summarizepaper.com/en/paper/1811.12808v1/

## Key points

- Proper techniques for model evaluation, model selection, and algorithm selection are crucial in machine learning.
- The article reviews different techniques for each of these subtasks and discusses their advantages and disadvantages.
- Recommendations are provided to promote best practices in machine learning research and applications.
- The limitations of common methods like the holdout method for model evaluation and selection when working with small datasets are highlighted.
- Various flavors of the bootstrap technique are introduced as an alternative for estimating performance uncertainty.
- Popular cross-validation techniques such as leave-one-out cross-validation and k-fold cross-validation are discussed, along with tips for selecting an optimal value of k based on empirical evidence.
- Different statistical tests for comparing algorithms are presented, along with strategies for dealing with multiple comparisons.
- Alternative methods for algorithm selection, such as combined F-test 5x2 cross validation and nested cross validation, are recommended when working with small datasets.

## Summary

In the field of machine learning, it is crucial to use proper techniques for model evaluation, model selection, and algorithm selection. This not only applies to academic research but also has significant implications in various industrial settings. In this article by Sebastian Raschka, different techniques for each of these subtasks are reviewed and their advantages and disadvantages are discussed based on theoretical and empirical studies. The article provides recommendations to promote best practices in machine learning research and applications. It highlights the limitations of common methods like the holdout method for model evaluation and selection when working with small datasets. Instead, it introduces various flavors of the bootstrap technique as an alternative for estimating performance uncertainty. These flavors can be used if bootstrapping is computationally feasible. Furthermore, the article delves into popular cross-validation techniques such as leave-one-out cross-validation and k-fold cross-validation. It discusses the bias-variance trade-off involved in choosing the value of k in k-fold cross-validation and offers practical tips for selecting an optimal value based on empirical evidence. The article also presents different statistical tests for comparing algorithms and explores strategies for dealing with multiple comparisons, including omnibus tests and multiple-comparison corrections. These approaches help researchers make reliable algorithm comparisons. Additionally, alternative methods for algorithm selection are recommended when working with small datasets; namely combined F-test 5x2 cross validation and nested cross validation which are suggested as effective approaches under such circumstances. Overall, this comprehensive review provides valuable insights into model evaluation, model selection, and algorithm selection techniques in machine learning research by considering both the advantages and disadvantages along with empirical evidence so that researchers can adopt best practices to enhance their work in this rapidly evolving field.