# When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

**Authors:** Biao Zhang, Zhongtao Liu, Colin Cherry, Orhan Firat  
**arXiv:** [2402.17193](https://arxiv.org/abs/2402.17193)  
**Date:** 2024-02-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.17193v1/

## Key points

- Study title: "When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method"
- Authors: Biao Zhang, Zhongtao Liu, Colin Cherry, Orhan Firat
- Investigated the impact of scaling factors on LLM finetuning performance
- Utilized pretrained bilingual LLMs ranging from 1B to 16B parameters
- Conducted experiments on machine translation and summarization benchmarks
- Observed power-based multiplicative joint scaling laws between finetuning data size and other factors
- Findings provide insights for effective LLM finetuning strategies in natural language processing technologies

## Summary

In their study titled "When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method," authors Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat delve into the intricate world of large language models (LLMs) and the finetuning methods used to enhance their performance for various downstream applications. The researchers conducted systematic experiments to investigate how different scaling factors impact LLM finetuning performance in scenarios where the size of the model exceeds available data. By utilizing pretrained bilingual LLMs ranging from 1B to 16B parameters and conducting tests on machine translation and summarization benchmarks, they made key observations about power-based multiplicative joint scaling laws between finetuning data size and other factors. Their findings highlight important considerations for understanding and developing effective LLM finetuning strategies in natural language processing technologies.