# Exploring the Effectiveness of Large Language Models in Generating Unit Tests

**Authors:** Mohammed Latif Siddiq, Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, Vinicius Carvalho Lopes  
**arXiv:** [2305.00418](https://arxiv.org/abs/2305.00418)  
**Date:** 2023-04-30  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2305.00418v1/

## Key points

- Study investigates the effectiveness of large language models in generating unit tests without fine-tuning
- Focus on three generative models: CodeGen, Codex, and GPT-3.5
- Two benchmarks used: HumanEval and Evosuite SF110
- Evaluation criteria include compilation rates, test correctness, coverage levels, and test smells
- Codex model achieves over 80% coverage for HumanEval dataset
- None of the models achieve more than 2% coverage for EvoSuite SF110 benchmark
- Generated tests exhibit test smells such as Duplicated Asserts and Empty Tests
- Study highlights areas where improvements are needed to enhance performance in generating unit tests

## Summary

In the study titled "Exploring the Effectiveness of Large Language Models in Generating Unit Tests," authors Mohammed Latif Siddiq, Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes investigate the potential of code generation models for generating unit tests without fine-tuning. They specifically focus on three generative models: CodeGen, Codex, and GPT-3.5. The researchers utilize two benchmarks to assess the impact of context generation on the unit test generation process: HumanEval and Evosuite SF110. The evaluation criteria include compilation rates, test correctness, coverage levels and test smells. The findings reveal that while the Codex model achieves over 80% coverage for the HumanEval dataset, none of the models achieve more than 2% coverage for the EvoSuite SF110 benchmark. Additionally, generated tests exhibit test smells such as Duplicated Asserts and Empty Tests. This study sheds light on the effectiveness of large language models in generating unit tests and highlights areas where improvements are needed to enhance their performance in this domain.