# ChatGPT as a tool for User Story Quality Evaluation: Trustworthy Out of the Box?

**Authors:** Krishna Ronanki, Beatriz Cabrero-Daniel, Christian Berger  
**arXiv:** [2306.12132](https://arxiv.org/abs/2306.12132)  
**Date:** 2023-06-21  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2306.12132v1/

## Key points

- ChatGPT, a general-purpose language model, used for evaluating user stories in Agile software development
- User stories capture end-user needs and facilitate communication within development teams
- ChatGPT's performance aligns well with human evaluation, compared to an existing benchmark
- "Best of three" strategy proposed to improve output stability and address concerns about trustworthiness and reliability in AI
- User stories commonly used in expressing requirements in Agile software development, following a specific template with elements such as role, goal, and benefit
- Few-shot prompting technique utilized to evaluate user story quality using ChatGPT
- Quality criteria presented by Lucassen et al. used for assessing individual user stories and sets of user stories
- T˜oemets' work on predicting the quality of user stories mentioned for monitoring purposes
- High agreement rates between human evaluations and ChatGPT's evaluations shown across various metrics
- Study demonstrates that ChatGPT can effectively evaluate the quality of user stories in Agile software development
- Provides insights into improving output stability of ChatGPT
- Highlights potential applications of AI in this domain while addressing concerns about trustworthiness and reliability.

## Summary

This study explores the use of ChatGPT, a general-purpose language model, for evaluating the quality of user stories in Agile software development. User stories are important for capturing end-user needs and facilitating communication within development teams. The researchers compare ChatGPT's performance with an existing benchmark and find that it aligns well with human evaluation. They propose a "best of three" strategy to improve output stability and address concerns about trustworthiness and reliability in AI. In the background section, the authors explain that user stories are commonly used in expressing requirements in Agile software development. They follow a specific template that includes elements such as role, goal, and benefit. The study utilizes few-shot prompting technique to evaluate user story quality using ChatGPT. This technique involves providing the model with a small number of examples as conditioning before asking it to evaluate user stories based on defined criteria. The researchers use the quality criteria presented by Lucassen et al., which consist of 13 criteria for assessing individual user stories and sets of user stories. They also mention T˜oemets' work on predicting the quality of user stories for monitoring purposes. The method section includes tables showing agreement rates between human evaluations and ChatGPT's evaluations using different interpretation strategies. The results show high agreement rates across various metrics. Overall, this study demonstrates that ChatGPT can effectively evaluate the quality of user stories in Agile software development and provides insights into improving its output stability. It highlights the potential applications of AI in this domain while addressing concerns about trustworthiness and reliability.