# Dataset Distillation using Neural Feature Regression

**Authors:** Yongchao Zhou, Ehsan Nezhadarya, Jimmy Ba  
**arXiv:** [2206.00719](https://arxiv.org/abs/2206.00719)  
**Date:** 2022-06-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.00719v1/

## Key points

- Dataset distillation aims to create a compact synthetic dataset retaining essential information from the original dataset.
- The process is framed as a bi-level meta-learning problem with outer loop optimizing the meta-dataset and inner loop training a model on distilled data.
- Authors propose neural Feature Regression with Pooling (FRePo) to address challenges like computing meta-gradients, reducing memory requirements, and faster training.
- FRePo operates similar to truncated backpropagation through time using a pool of models to mitigate overfitting in dataset distillation tasks.
- FRePo outperforms existing methods significantly on benchmark datasets such as CIFAR100, Tiny ImageNet, and ImageNet-1K.
- High-quality distilled data can enhance downstream applications like continual learning and membership inference defense.

## Summary

In their paper titled "Dataset Distillation using Neural Feature Regression," authors Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba delve into the realm of dataset distillation. This process aims to create a compact synthetic dataset that retains essential information from the original dataset. The task is framed as a bi-level meta-learning problem, with the outer loop optimizing the meta-dataset and the inner loop involving training a model on the distilled data. A key challenge in this approach lies in computing meta-gradients. Differentiating through the inner loop learning procedure incurs significant computational and memory costs. To tackle these challenges, the authors propose a novel solution called neural Feature Regression with Pooling (FRePo). This method not only achieves state-of-the-art performance but also boasts an order of magnitude reduction in memory requirements and two orders of magnitude faster training compared to previous techniques. FRePo operates akin to truncated backpropagation through time by utilizing a pool of models to mitigate various forms of overfitting commonly encountered in dataset distillation tasks. The effectiveness of FRePo is demonstrated through extensive experiments on benchmark datasets such as CIFAR100, Tiny ImageNet, and ImageNet-1K. It outperforms existing methods significantly. Moreover, the authors showcase how high-quality distilled data can greatly enhance downstream applications like continual learning and membership inference defense. In conclusion, "Dataset Distillation using Neural Feature Regression" presents a cutting-edge approach to dataset distillation that not only addresses key challenges in meta-gradient computation but also showcases superior performance across various datasets and downstream tasks.