# The Diversity of Gamification Evaluation in the Software Engineering Education and Industry: Trends, Comparisons and Gaps

**Authors:** Rodrigo Henrique Barbosa Monteiro, Maurício Ronny de Almeida Souza, Sandro Ronaldo Bezerra Oliveira, Carlos dos Santos Portela, Cesar Elias de Cristo Lobato  
**arXiv:** [2102.05089](https://arxiv.org/abs/2102.05089)  
**Date:** 2021-02-09  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2102.05089v1/

## Key points

- Gamification is a promising approach to motivate and engage participants in software engineering education and practice activities.
- A systematic mapping study was conducted to investigate strategies for evaluating gamification in the context of software engineering.
- The study selected 100 primary studies on gamification in software engineering from 2011 to 2020.
- The results indicate that while 64 studies report procedures for evaluating gamification, only three propose evaluation models.
- The evaluation of gamification focuses on two aspects: evaluating the gamification strategy itself related to user experience and perceptions; and evaluating its outcomes and effects on users and context.
- The most recurring criteria used in evaluations are engagement, motivation, satisfaction, and performance.
- Evaluation requires a mix of subjective and objective inputs with qualitative and quantitative data analysis approaches.
- Depending on whether the focus is on strategy or outcomes evaluation, there is a predominance of certain types of data collection methods or analysis techniques.
- This study highlights the need for standardization in evaluating gamification approaches in software engineering education and practice activities.

## Summary

Gamification has emerged as a promising approach to motivate and engage participants in software engineering education and practice activities. A systematic mapping study was conducted to investigate strategies for evaluating gamification in the context of software engineering. The study selected 100 primary studies on gamification in software engineering from 2011 to 2020. These studies were categorized based on the presence of evaluation procedures or models for gamification, the purpose of evaluation, criteria used, type of data collected, instruments used, and procedures for data analysis. The results indicate that while 64 studies report procedures for evaluating gamification, only three propose evaluation models. The evaluation of gamification focuses on two aspects: evaluating the gamification strategy itself related to user experience and perceptions; and evaluating its outcomes and effects on users and context. The most recurring criteria used in evaluations are engagement, motivation, satisfaction, and performance. Evaluation requires a mix of subjective and objective inputs with qualitative and quantitative data analysis approaches. Depending on whether the focus is on strategy or outcomes evaluation, there is a predominance of certain types of data collection methods or analysis techniques. For instance, when evaluating strategy-related aspects such as user experience or perception towards gamified systems or applications using surveys or interviews may be more appropriate than analyzing performance metrics. In conclusion, this study highlights the need for standardization in evaluating gamification approaches in software engineering education and practice activities. It also provides insights into different strategies employed by researchers in this field while identifying gaps that require further investigation.