# CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

**Authors:** Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, Yingbo Zhou  
**arXiv:** [2305.02309](https://arxiv.org/abs/2305.02309)  
**Date:** 2023-05-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.02309v1/

## Key points

- Large language models (LLMs) are effective in representation learning for program synthesis and comprehension tasks.
- The quality of learned representations is limited by neural scaling laws and upper bounds on model performance imposed by available data and compute.
- A team of researchers conducted a study to make LLM training for program synthesis more efficient by unifying four key components: model architectures, learning methods, infill sampling, and data distributions.
- They unified encoder and decoder-based models into a single prefix-LM for the model architecture.
- For learning methods, they unified causal language modeling, span corruption, and infilling into a simple learning algorithm.
- They explored the "free lunch" hypothesis for infill sampling and investigated the effect of programming and natural language mixture distribution on model performance for data distributions.
- The research team distilled failures and successes into four lessons to provide a final recipe for training CodeGen2 models with sizes ranging from 1B to 16B parameters along with an open-source training framework available at https://github.com/salesforce/CodeGen2.
- This study provides insights into making LLM training more efficient for program synthesis tasks by unifying key components such as model architectures and learning methods while exploring infill sampling techniques and data distributions' effects on performance.

## Summary

Large language models (LLMs) have shown impressive capabilities in representation learning for program synthesis and comprehension tasks. However, the quality of learned representations is limited by neural scaling laws that depend on the number of model parameters and observations, while upper bounds on model performance are imposed by available data and compute which can be expensive. To address this issue, a team of researchers led by Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese and Yingbo Zhou conducted a study to make LLM training for program synthesis more efficient. The study aimed to unify four key components: model architectures, learning methods, infill sampling and data distributions. The researchers unified encoder and decoder-based models into a single prefix-LM for the model architecture. For learning methods they unified causal language modeling, span corruption and infilling into a simple learning algorithm. They also explored the "free lunch" hypothesis for infill sampling and investigated the effect of programming and natural language mixture distribution on model performance for data distributions. The research team conducted comprehensive empirical experiments on 1B LLMs to distill failures and successes into four lessons. They provide a final recipe for training CodeGen2 models with sizes ranging from 1B to 16B parameters along with an open-source training framework available at https://github.com/salesforce/CodeGen2. Overall this study provides insights into making LLM training more efficient for program synthesis tasks by unifying key components such as model architectures and learning methods while exploring infill sampling techniques and data distributions' effects on performance. These findings could have significant implications in advancing natural language processing technologies further.