# Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning

**Authors:** Chao-Chung Wu, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee  
**arXiv:** [2501.14315](https://arxiv.org/abs/2501.14315)  
**Date:** 2025-01-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.14315v6/

## Key points

- Researchers investigate the use of Language Model (LLM)-generated data for fine-tuning in machine learning and its impact on cross-domain generalization.
- Using LLM-generated data enhances target task performance and reduces degradation in non-target tasks compared to using ground truth data.
- Reduction of high perplexity tokens in LLM-generated sequences is key to achieving improved performance, as demonstrated across different model families and scales.
- Two methods for generating training data - Self-Output and Rephrase - are explored for two target tasks, with evaluation on five non-target tasks.
- Detailed information is provided on original datasets used for data generation and methodology for constructing self-generated training datasets using language models.
- The research offers valuable insights into leveraging LLM-generated data for fine-tuning to improve model performance across domains and mitigate forgetting issues in machine learning applications.

## Summary

In this study, the researchers investigate the use of Language Model (LLM)-generated data for fine-tuning in machine learning and its impact on cross-domain generalization. They conduct a systematic analysis to reveal that using LLM-generated data not only enhances target task performance but also reduces degradation in non-target tasks compared to using ground truth data. This is achieved by reducing high perplexity tokens present in LLM-generated sequences, as shown through experiments across different model families and scales. The study also explores two methods for generating training data - Self-Output and Rephrase - for two different target tasks and evaluates their effectiveness on five non-target tasks. Additionally, detailed information is provided on the original datasets used for data generation and the methodology for constructing self-generated training datasets using language models. Overall, this research offers valuable insights into leveraging LLM-generated data for fine-tuning to improve model performance across domains and mitigate forgetting issues commonly observed in machine learning applications.