# Code Llama: Open Foundation Models for Code

**Authors:** Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar  
**arXiv:** [2308.12950](https://arxiv.org/abs/2308.12950)  
**Date:** 2023-08-24  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2308.12950v2/

## Key points

- Code Llama is a family of large language models for code
- Offers state-of-the-art performance, infilling capabilities, support for large input contexts, and the ability to follow instructions for programming tasks
- Available in multiple flavors including foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct)
- All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens
- Introduces a dedicated long context fine-tuning (LCFT) stage to handle long sequences effectively
- LCFT stage modifies the rotation frequencies of the rotary position embedding used in the foundation model
- Instruction fine-tuned models (Code Llama - Instruct) are based on Code Llama and trained to answer questions appropriately
- Trained on a proprietary dataset that combines supervised fine-tuning and rejection sampling examples collected through reinforcement learning from human feedback
- Capable of performing various code understanding and synthesis tasks such as code summarization, refinement, translation, bug fixing, build error fixing, and solving math problems
- Outperforms other publicly available models on various benchmarks
- Released under a permissive license allowing for both research and commercial use.

## Summary

Code Llama is a family of large language models for code that offers state-of-the-art performance, infilling capabilities, support for large input contexts and the ability to follow instructions for programming tasks. The models are available in multiple flavors including foundation models (Code Llama), Python specializations (Code Llama - Python) and instruction-following models (Code Llama - Instruct) with varying parameters. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. To effectively handle long sequences, Code Llama introduces a dedicated long context fine-tuning (LCFT) stage. During this stage, the models are trained by modifying the rotation frequencies of the rotary position embedding used in the foundation model. By increasing the base period from 10k to 1 million for fine-tuning, Code Llama gains long-range capabilities and reduces bias towards short-distance attention. The instruction fine-tuned models (Code Llama - Instruct) are based on Code Llama and trained to answer questions appropriately. They are trained on a proprietary dataset that combines supervised fine-tuning and rejection sampling examples collected through reinforcement learning from human feedback. This dataset enables Code Llama to inherit instruction following and safety properties from its predecessor model, Llama 2. In addition to program synthesis and infilling tasks, Code Llama is also capable of performing various code understanding and synthesis tasks such as code summarization, refinement, translation, bug fixing, build error fixing as well as solving math problems. Overall, Code Llama provides advanced capabilities for code related tasks and outperforms other publicly available models on various benchmarks. It is released under a permissive license that allows for both research and commercial use.