Meta-Learning a Real-Time Tabular AutoML Method For Small Data

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Not relevant

   Towards efficient student performance prediction with tabular prior-data fitted network: a small-sample learning approach in data-scarce scenarios

   Pei-Han Jiang, Xuan-Ping Wu, Wen-Ting Wang et al. · 2026 · PeerJ Computer Science · Not summarized yet

   Abstract

   Accurate prediction of student academic performance is crucial for enhancing educational outcomes and enabling personalized learning interventions. However, the limited availability of labeled data in many practical educational settings poses significant challenges for conventional machine learning models, which often require large datasets to achieve satisfactory performance and are prone to overfitting. To address this issue, we propose a novel small-sample learning framework based on the Tabular Prior-Data Fitted Network (TabPFN). By integrating a Transformer based architecture with Bayesian inference, TabPFN effectively leverages meta-learned priors to perform high-quality predictions wi…

2. Very close Not relevant

   Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data

   Anurag Garg, Muhammad Ali, N. Hollmann et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 56 times · Not summarized yet

   Abstract

   Foundation models for tabular data, like TabPFN, achieve strong performance on small datasets when pre-trained solely on synthetic data. We show that this performance can be significantly boosted by a targeted continued pre-training phase. Specifically, we demonstrate that leveraging a small, curated collection of large, real-world datasets for continued pre-training yields superior downstream predictive accuracy compared to using broader, potentially noisier corpora like CommonCrawl or GitTables. Our resulting model, Real-TabPFN, achieves substantial performance gains on 29 datasets from the OpenML AutoML Benchmark.

3. Very close Not relevant

   Transformer-Based Tabular Foundation Models: Outperforming Traditional Methods with TabPFN

   R. Babu, Vishwa Priya V, Manoj Kumar Mishra et al. · 2025 · International Journal of Engineering Science and Information Technology · cited 2 times · Not summarized yet

   Abstract

   Scientific research and commercial applications rely heavily on tabular data, yet efficiently modelling this data has constantly been a problem. For over twenty years, the standard method for machine learning has been based on traditional models, with gradient-boosted decision trees (GBDTs). Despite recent advancements in deep learning, neural networks often fail to provide satisfactory results on compact tabular datasets due to factors such as overfitting, insufficient data intricate feature relationships. The study offers a Tabular Prior data Fitted Network, a foundation model developed by meta-learning on more than one million synthetic datasets generated sequentially, which is constructe…

4. Very close Not relevant

   nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN

   Alexander Pfefferle, Johannes Hog, L. Purucker et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 19 times · Not summarized yet

   Abstract

   Tabular foundation models such as TabPFN have revolutionized predictive machine learning for tabular data. At the same time, the driving factors of this revolution are hard to understand. Existing open-source tabular foundation models are implemented in complicated pipelines boasting over 10,000 lines of code, lack architecture documentation or code quality. In short, the implementations are hard to understand, not beginner-friendly, and complicated to adapt for new experiments. We introduce nanoTabPFN, a simplified and lightweight implementation of the TabPFN v2 architecture and a corresponding training loop that uses pre-generated training data. nanoTabPFN makes tabular foundation models m…

5. Very close Not relevant

   Accurate predictions on small data with a tabular foundation model

   N. Hollmann, Samuel Müller, L. Purucker et al. · 2025 · Nature · cited 1,756 times · Not summarized yet

   Abstract

   Tabular data, spreadsheets organized in rows and columns, are ubiquitous across scientific fields, from biomedicine to particle physics to economics and climate science1,2. The fundamental prediction task of filling in missing values of a label column based on the rest of the columns is essential for various applications as diverse as biomedical risk models, drug discovery and materials science. Although deep learning has revolutionized learning from raw data and led to numerous high-profile success stories3, 4–5, gradient-boosted decision trees6, 7, 8–9 have dominated tabular data for the past 20 years. Here we present the Tabular Prior-data Fitted Network (TabPFN), a tabular foundation mod…

6. Close Not relevant

   Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data

   Kai Helli, David Schnurr, N. Hollmann et al. · 2024 · Neural Information Processing Systems · cited 49 times · Not summarized yet

   Abstract

   While most ML models expect independent and identically distributed data, this assumption is often violated in real-world scenarios due to distribution shifts, resulting in the degradation of machine learning model performance. Until now, no tabular method has consistently outperformed classical supervised learning, which ignores these shifts. To address temporal distribution shifts, we present Drift-Resilient TabPFN, a fresh approach based on In-Context Learning with a Prior-Data Fitted Network that learns the learning algorithm itself: it accepts the entire training dataset as input and makes predictions on the test set in a single forward pass. Specifically, it learns to approximate Bayes…

7. Close Not relevant

   Retrieval & Fine-Tuning for In-Context Tabular Models

   Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh et al. · 2024 · Neural Information Processing Systems · cited 81 times · Not summarized yet

   Abstract

   Tabular data is a pervasive modality spanning a wide range of domains, and the inherent diversity poses a considerable challenge for deep learning. Recent advancements using transformer-based in-context learning have shown promise on smaller and less complex datasets, but have struggled to scale to larger and more complex ones. To address this limitation, we propose a combination of retrieval and fine-tuning: we can adapt the transformer to a local subset of the data by collecting nearest neighbours, and then perform task-specific fine-tuning with this retrieved set of neighbours in context. Using TabPFN as the base model -- currently the best tabular in-context learner -- and applying our r…

8. Close Not relevant

   In-Context Data Distillation with TabPFN

   Junwei Ma, Valentin Thomas, Guangwei Yu et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 22 times · Not summarized yet

   Abstract

   Foundation models have revolutionized tasks in computer vision and natural language processing. However, in the realm of tabular data, tree-based models like XGBoost continue to dominate. TabPFN, a transformer model tailored for tabular data, mirrors recent foundation models in its exceptional in-context learning capability, being competitive with XGBoost's performance without the need for task-specific training or hyperparameter tuning. Despite its promise, TabPFN's applicability is hindered by its data size constraint, limiting its use in real-world scenarios. To address this, we present in-context data distillation (ICD), a novel methodology that effectively eliminates these constraints b…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data

   Nick Erickson, Jonas W. Mueller, Alexander Shirkov et al. · 2020 · arXiv.org · preprint, not peer reviewed · cited 1,126 times · Not summarized yet

   Abstract

   We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models on an unprocessed tabular dataset such as a CSV file. Unlike existing AutoML frameworks that primarily focus on model/hyperparameter selection, AutoGluon-Tabular succeeds by ensembling multiple models and stacking them in multiple layers. Experiments reveal that our multi-layer combination of many models offers better use of allocated training time than seeking out the best. A second contribution is an extensive evaluation of public and commercial AutoML platforms including TPOT, H2O, AutoWEKA, auto-sklearn, AutoGluon, and Google AutoML T…

2. Close Used in depth by this paper A method it uses Not relevant

   Why do tree-based models still outperform deep learning on tabular data?

   Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux · 2022 · Machine Learning · preprint, not peer reviewed · cited 709 times · AI summary available

   Abstract

   While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art on medium-sized data ($\sim$10K samples) even without accounting for their superior speed. To unders…

3. Close Used in depth by this paper A method it uses Not relevant

   SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

   Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild et al. · 2021 · Machine Learning · preprint, not peer reviewed · cited 639 times · AI summary available

   Abstract

   Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance ove…

4. Close Used in depth by this paper A method it uses Not relevant

   Well-tuned Simple Nets Excel on Tabular Datasets

   Arlind Kadra, M. Lindauer, F. Hutter et al. · 2021 · Neural Information Processing Systems · cited 294 times · Not summarized yet

   Abstract

   Tabular datasets are the last"unconquered castle"for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using a joint optimization over the decision on which regularizers to apply and their subsidiary hyperparame…

5. Related Used in depth by this paper A method it uses Not relevant

   Transformers Can Do Bayesian Inference

   Samuel Müller, N. Hollmann, Sebastian Pineda-Arango et al. · 2021 · International Conference on Learning Representations · cited 467 times · Not summarized yet

   Abstract

   Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification of prior knowledge and accurately capture model uncertainty. We present Prior-Data Fitted Networks (PFNs). PFNs leverage in-context learning in large-scale machine learning techniques to approximate a large set of posteriors. The only requirement for PFNs to work is the ability to sample from a prior distribution over supervised learning tasks (or functions). Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and thei…

Papers that cite it

1. Close Uses this paper in depth Not relevant

   TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models

   Deqing Fu, Huang-Yuan Su, Rajat Sen et al. · 2026 · cited 2 times · Not summarized yet

   Abstract

   Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, task descriptions, and auxiliary files that carry dataset semantics. Meanwhile, self-evolving machine learning engineering (MLE) agents train models from scratch on each dataset, yet jointly searching over features, architectures, and hyperparameters is noisy and prone to overfitting. We introduce TabFM-Auto, which pairs a tabular foundation model, TabFM, with a language model agent that evolves the data pipeline around it. Guided by dataset metadata and validation feedback, TabFM-Auto iteratively refines data cleaning, feature engineering, cont…

2. Close Uses this paper in depth Not relevant

   TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

   Rasa Hosseinzadeh, Alex Labach, Zexin Xue et al. · 2026 · cited 10 times · Not summarized yet

   Abstract

   Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on Tab…

3. Close Uses this paper in depth Not relevant

   Performance Evaluation of TabPFN for Student Depression Prediction Across Varying Sample Sizes

   F. Y. Arini, Muhammad Kahvi Khakam Syah, F. Setiawan et al. · 2026 · Jurnal Ilmu Komputer dan Informasi · Not summarized yet

   Abstract

   Early prediction of depression in students is a critical challenge, often hindered by the scarcity of large, labelled datasets. While supervised tabular classifiers such as Random Forest, XGBoost, and CatBoost are powerful, they typically require sufficient data and careful hyperparameter optimisation (HPO) to generalise effectively. This paper evaluates the Tabular Prior-data Fitted Network (TabPFN), a foundation model for supervised tabular learning, as a zero-shot classifier for student depression prediction. We conduct a comparative study against five robustly configured baseline classifiers (Random Forest, XGBoost, CatBoost, SVM, and Naive Bayes) across three publicly available student …

4. Close Not relevant

   TabPFN-3.5: Technical Report

   Benjamin Jäger, Nick Erickson, Léo Grinsztajn et al. · 2026 · cited 7 times · Not summarized yet

   Abstract

   We introduce TabPFN-3.5, our new flagship Tabular Foundation Model. It significantly outperforms its predecessor, TabPFN-3, and all existing baselines across a broad range of tabular problems. TabPFN-3.5 sets a new state of the art on standard tabular prediction in TabArena, and extends it to the data practitioners encounter in practice: non-i.i.d. data with temporal or grouped splits, tables with strings, text and images, high-cardinality categorical features, and wide tables with many features. These gains carry over to our task-specific harnesses: state of the art on relational data and stronger time-series forecasting. For faster inference, our variant TabPFN-3.5-Fast runs up to 3x faste…

5. Close Uses this paper in depth Not relevant

   TAROT: Task-Adaptive Refinement of LLM-prior Graphs for Few-shot Tabular Learning

   Ru-Xue Shi, Yi-Li Wang, Mengnan Du et al. · 2026 · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · Not summarized yet

   Abstract

   Few-shot tabular learning provides a cost-effective approach for real-world applications where annotation is costly and collecting sufficient samples for new tasks is difficult. Existing Traditional and LLM-based methods have demonstrated effectiveness in few-shot scenarios. However, traditional methods need additional training on unlabeled or generated data, which incur significant computational overhead. In addition, LLM-based methods that directly feed raw tabular data into LLMs raise privacy and compliance concerns. More importantly, both paradigms largely overlook the semantic relationships between features, which provide structural and semantic prior for constructing a semantic graph. …