# A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It

**Authors:** Martin Gruber, Gordon Fraser  
**arXiv:** [2203.00483](https://arxiv.org/abs/2203.00483)  
**Date:** 2022-03-01  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2203.00483v1/

## Key points

- Flaky tests, non-deterministically passing and failing test cases, have become a significant issue in software engineering.
- Martin Gruber and Gordon Fraser conducted a survey involving 335 professional software developers and testers, revealing that flaky tests are prevalent and serious.
- Developers are more concerned about losing trust in test outcomes than the computational costs of re-running tests.
- Addressing flakiness requires both technical solutions and consideration of psychological aspects.
- Developers expressed a need for support tools like IDE plugins for early detection of flakiness and visualizations such as dashboards displaying test outcomes over time.
- There is a desire for more training and information on effectively dealing with flakiness among developers.
- Researchers and tool developers play a critical role in improving detection methods, providing better visualization tools, and offering educational resources to enhance the reliability of software testing processes.

## Summary

In recent years, the issue of non-deterministically passing and failing test cases, known as flaky tests, has garnered significant attention in the field of software engineering. Previous research primarily focused on analyzing flakiness through a code-centric approach by mining software repositories. However, there has been a growing interest in understanding how developers perceive and experience flakiness in their daily work. To delve deeper into this aspect, Martin Gruber and Gordon Fraser conducted a survey involving 335 professional software developers and testers across various domains. The results of the survey highlighted that flaky tests are indeed a prevalent and serious problem faced by developers. Contrary to common assumptions, developers expressed less concern about the computational costs associated with re-running tests and more about the implications of losing trust in the test outcomes. This finding underscores the importance of addressing flakiness not just from a technical standpoint but also from a psychological perspective. In response to these challenges, developers expressed a clear need for support tools that can help them detect flaky code effectively. Specifically, they emphasized the importance of IDE plugins for identifying flakiness early on and visualizations such as dashboards displaying test outcomes over time. Additionally, developers expressed a desire for more training and information on dealing with flakiness effectively. Overall, the survey findings underscore the critical role that researchers and tool developers play in addressing the issue of test flakiness. By focusing on improving detection methods, providing better visualization tools, and offering educational resources on mitigating flakiness, stakeholders can work towards enhancing the reliability and trustworthiness of software testing processes in real-world development environments.