# Code Execution with Pre-trained Language Models

**Authors:** Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, Nan Duan  
**arXiv:** [2305.05383](https://arxiv.org/abs/2305.05383)  
**Date:** 2023-05-08  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2305.05383v1/

## Key points

- Authors explore importance of code execution in programming language semantics
- Existing pre-trained models for code intelligence focus on source code and syntax, ignoring execution trace
- Authors develop mutation-based data augmentation technique to create realistic Python dataset for code execution
- Dataset challenges existing models like Codex
- Introduce CodeExecutor, a Transformer model that leverages code execution pre-training and curriculum learning
- Evaluate CodeExecutor's performance on code execution tasks and demonstrate promising results as well as limitations
- Potential benefits of CodeExecutor for other code intelligence tasks such as zero-shot code-to-code search and text-to-code generation
- Study provides insights into learning and generalization abilities of pre-trained models for code execution
- Incorporating execution trace aims to improve understanding and performance of pre-trained models in handling complex programming tasks.

## Summary

In the paper titled "Code Execution with Pre-trained Language Models," authors Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan explore the importance of code execution in programming language semantics. They highlight that most pre-trained models for code intelligence focus solely on source code and syntactic structures while ignoring the execution trace. To address this limitation, the authors develop a mutation-based data augmentation technique to create a realistic Python dataset for code execution. This dataset challenges existing models like Codex. Additionally, they introduce CodeExecutor, a Transformer model that leverages code execution pre-training and curriculum learning to enhance its semantic comprehension. The authors evaluate CodeExecutor's performance on code execution tasks and demonstrate its promising results as well as its limitations. Furthermore, they showcase the potential benefits of CodeExecutor for other code intelligence tasks such as zero-shot code-to-code search and text-to-code generation. Overall, this study provides valuable insights into the learning and generalization abilities of pre-trained models for code execution. By incorporating the execution trace into their approach, the authors aim to improve the understanding and performance of pre-trained models in handling complex programming tasks.