# Why do tree-based models still outperform deep learning on tabular data?

**Authors:** Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux  
**arXiv:** [2207.08815](https://arxiv.org/abs/2207.08815)  
**Date:** 2022-07-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2207.08815v1/

## Key points

- Authors conducted a comprehensive analysis comparing deep learning models to tree-based models on tabular data
- Tree-based models outperformed deep learning models on medium-sized datasets containing around 10,000 samples
- Identified key challenges for neural networks in handling tabular data: robustness against uninformative features, preservation of data orientation, and effective learning of irregular functions
- Provided a standardized benchmark and raw data from an extensive hyperparameter search to encourage further research in developing specialized neural network architectures for tabular data

## Summary

In their paper titled "Why do tree-based models still outperform deep learning on tabular data? ", authors Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux delve into the ongoing debate surrounding the performance of deep learning models compared to traditional tree-based models on tabular data. While deep learning has shown remarkable success in text and image datasets, its effectiveness in handling tabular data remains uncertain. To address this issue, the authors conducted a comprehensive analysis by benchmarking a wide range of standard and innovative deep learning techniques alongside well-established tree-based models like XGBoost and Random Forests. They utilized a diverse set of 45 datasets from various domains specifically chosen for their tabular characteristics. The benchmarking methodology employed accounted for both model fitting and hyperparameter optimization across different dataset sizes and complexities. Surprisingly, the results revealed that tree-based models continued to outperform deep learning models on medium-sized datasets containing approximately 10,000 samples, even without considering their superior computational efficiency. To gain insights into this performance gap, the authors conducted an empirical investigation into the inherent biases of tree-based models versus Neural Networks (NNs). This analysis led to the identification of key challenges that researchers should consider when designing neural networks tailored for tabular data. These challenges include the need for robustness against uninformative features, preservation of data orientation, and the ability to effectively learn irregular functions. To encourage further research in developing specialized neural network architectures for tabular data, the authors provided a standardized benchmark along with raw data from an extensive 20,000 compute hours hyperparameter search for each learner. Overall, this study sheds light on the factors contributing to the continued dominance of tree-based models over deep learning approaches in handling tabular data and provides valuable insights for future research directions in this field.