# Unit Test Case Generation with Transformers

**Authors:** Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, Neel Sundaresan  
**arXiv:** [2009.05617](https://arxiv.org/abs/2009.05617)  
**Date:** 2020-09-11  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2009.05617v1/

## Key points

- Existing approaches in software testing for automated unit test case generation rely on test coverage criteria and generate synthetic test cases that are challenging for developers to comprehend
- AthenaTest is a novel approach that leverages real-world, developer-written test cases to generate unit test cases
- AthenaTest utilizes a state-of-the-art sequence-to-sequence transformer model
- Methods2test is the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java
- The authors use the methods2test dataset to train the transformer model by translating focal methods into their corresponding test cases
- The effectiveness of the model is evaluated based on natural language processing and code-specific criteria, including quality of translation, syntactic correctness, presence of testing APIs, and coverage information
- Comparison is made with EvoSuite and GPT-3 generated test cases
- Feedback from professional developers through a survey indicates positive results in terms of readability, understandability, and testing effectiveness of the generated test cases

## Summary

In the field of software testing, there has been a significant amount of research on automated unit test case generation. However, existing approaches often rely on test coverage criteria and generate synthetic test cases that are challenging for developers to comprehend. To address this issue, the authors propose AthenaTest, a novel approach that leverages real-world, developer-written test cases to generate unit test cases. AthenaTest utilizes a state-of-the-art sequence-to-sequence transformer model, which is capable of generating useful test cases for a specific method under test (referred to as the focal method). The authors also introduce methods2test, which is the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java. This corpus consists of 630k test cases extracted from 70k open-source repositories on GitHub. To train the transformer model, the authors use the methods2test dataset to translate focal methods into their corresponding test cases. They evaluate the effectiveness of their model by considering both natural language processing and code-specific criteria. Firstly, they assess the quality of the translation compared to the target test case. Then, they analyze various properties of the generated test cases such as syntactic correctness and the presence of different testing APIs (e.g., asserts). To further validate their approach, the authors execute the generated test cases and collect information about their coverage. They compare these results with those obtained from test cases generated by EvoSuite and GPT-3. Additionally, they conduct a survey among professional developers to gather feedback on readability, understandability, and testing effectiveness of the generated test cases. Overall, this paper presents AthenaTest as an innovative approach for automated unit test case generation that addresses some limitations of existing techniques. The authors provide extensive evaluation results and insights from professional developers to demonstrate its potential benefits in terms of generating high-quality and developer-friendly unit tests.