# Improved Techniques for Training Consistency Models

**Authors:** Yang Song, Prafulla Dhariwal  
**arXiv:** [2310.14189](https://arxiv.org/abs/2310.14189)  
**Date:** 2023-10-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.14189v1/

## Key points

- Consistency models in generative modeling are a promising approach for high-quality data generation without adversarial training
- Existing methods face limitations like reliance on pre-trained models and bias in evaluation metrics like LPIPS
- Recent study by Yang Song and Prafulla Dhariwal introduces advanced techniques for training consistency models, including eliminating Exponential Moving Average from teacher consistency model
- Proposed method allows consistency models to learn directly from data, enhancing their ability to generate high-quality samples independently
- Researchers leverage Pseudo-Huber losses from robust statistics to replace biased metrics like LPIPS, improving evaluation process and overall performance of consistency models
- Introduction of lognormal noise schedule and strategy to double total discretization steps at regular intervals during training iterations enhances performance of consistency models
- Refined consistency models achieve remarkable results on benchmark datasets with FID scores of 2.51 and 3.25 on CIFAR-10 and ImageNet $64\times 64$ respectively in one sampling step, showcasing significant improvement in sample quality compared to previous methods
- Two-step sampling strategies further reduce FID scores to 2.24 and 2.77 on these datasets, surpassing distillation-based results while narrowing the performance gap with state-of-the-art generative models

## Summary

, , , , 

In the realm of generative models, consistency models have emerged as a promising approach to generating high-quality data in a single step without the need for adversarial training. These models have shown great potential by distilling knowledge from pre-trained diffusion models and utilizing metrics like LPIPS to achieve optimal sample quality. However, existing methods face limitations such as being constrained by the quality of the pre-trained model and introducing bias in evaluation through metrics like LPIPS. To address these challenges, a recent study by Yang Song and Prafulla Dhariwal introduces advanced techniques for training consistency models. One key innovation is the elimination of Exponential Moving Average from the teacher consistency model, which was identified as a previously overlooked flaw in traditional approaches. Instead of relying on distillation, the proposed method allows consistency models to learn directly from data, thereby enhancing their ability to generate high-quality samples independently. Moreover, to replace biased metrics like LPIPS, the researchers leverage Pseudo-Huber losses from robust statistics. This adjustment not only improves the evaluation process but also enhances the overall performance of consistency models. Additionally, a lognormal noise schedule is introduced for the consistency training objective, along with a strategy to double total discretization steps at regular intervals during training iterations. Through meticulous hyperparameter tuning and these novel techniques, the refined consistency models achieve remarkable results on benchmark datasets. In particular, they attain FID scores of 2.51 and 3.25 on CIFAR-10 and ImageNet $64\times 64$ respectively in just one sampling step. These scores represent a significant improvement compared to previous methods, showcasing a 3.5$\times$ and 4$\times$ enhancement in sample quality. Furthermore, by implementing two-step sampling strategies, FID scores are further reduced to 2.24 and 2.77 on these datasets. Notably, these results surpass those obtained through distillation in both one-step and two-step settings while narrowing the performance gap between consistency models and other state-of-the-art generative models. In conclusion, this research presents cutting-edge advancements in training consistency models that pave the way for more efficient and effective data generation processes within the field of generative modeling.