# Automated Unit Test Improvement using Large Language Models at Meta

**Authors:** Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, Eddy Wang  
**arXiv:** [2402.09171](https://arxiv.org/abs/2402.09171)  
**Date:** 2024-02-14  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2402.09171v1/

## Key points

- Meta's TestGen-LLM tool utilizes Large Language Models (LLMs) to enhance human-written tests automatically
- Generated test classes pass filters for significant improvements over original test suite and to mitigate LLM hallucination issues
- Deployment of TestGen-LLM at Meta test-a-thons for Instagram and Facebook platforms shows promising results
- Evaluation focusing on Reels and Stories products for Instagram: 75% of TestGen-LLM's test cases successfully built, 57% passed reliably, resulting in 25% coverage increase
- Diff-time deployment mode provides engineers with full context of testing and code under review, showcasing effectiveness in real-world scenarios
- Construction of TestGen-LLM diffs during Instagram Test-a-thons yielded promising results, with some diffs significantly improving coverage by covering previously untouched methods and files
- Previous literature reviews confirm prevalence of LLM-based test generation approaches; this paper stands out for extending existing test classes and reporting industrial-scale deployment results

## Summary

This paper presents Meta's TestGen-LLM tool, which utilizes Large Language Models (LLMs) to automatically enhance human-written tests. The tool ensures that the generated test classes pass a set of filters to guarantee significant improvements over the original test suite and mitigate issues related to LLM hallucination. The deployment of TestGen-LLM at Meta test-a-thons for Instagram and Facebook platforms is discussed, showcasing promising results. In an evaluation focusing on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases were successfully built and 57% passed reliably, resulting in a 25% increase in coverage. The paper emphasizes the effectiveness of deploying tests at diff time as it provides engineers with full context of existing testing and code under review. Insights into diff-time deployment mode were obtained through experiences gained from a test-a-thon, shedding light on how this technology performs in real-world scenarios. Initially done manually but later automated in subsequent events, the construction of TestGen-LLM diffs for the Instagram Test-a-thons yielded promising results. During the first Instagram Test-a-thon, 36 engineers landed 105 unit test diffs with 16 generated by TestGen-LLM. Notably, one diff was rejected due to lack of assertion in the test case. The outcomes varied with some diffs significantly improving coverage by covering previously untouched methods and files. The largest coverage improvement stemmed from a diff that covered multiple new files and A/B testing gatekeepers. In terms of related work, software test generation within the realm of Large Language Model-based Software Engineering (LLMSE) has been extensively studied. While previous literature reviews confirm the prevalence of LLM-based test generation approaches, this paper stands out for its focus on extending existing test classes and reporting results from industrial-scale deployment. Overall, this paper contributes valuable insights into automated unit test improvement using LLMs at Meta through diff-time deployment strategies and showcases promising results from real-world applications on popular social media platforms like Instagram and Facebook.