# Human-AI Collaboration for UX Evaluation: Effects of Explanation and Synchronization

**Authors:** Mingming Fan, Xianyou Yang, Tsz Tung Yu, Vera Q. Liao, Jian Zhao  
**arXiv:** [2112.12387](https://arxiv.org/abs/2112.12387)  
**Date:** 2021-12-23  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2112.12387v1/

## Key points

- Study focuses on designing AI systems for collaboration between UX evaluators and AI in analyzing usability test videos
- Two key design factors investigated: explanations and synchronization
- Explanations involve AI informing humans about how it identifies UX problems during usability tests
- Synchronization refers to the ways humans and AI collaborate (synchronously or asynchronously)
- Developed a tool called AI Assistant with four versions of user interfaces representing different levels of explanations and synchronization
- Used a hybrid wizard-of-oz approach to simulate AI performance
- Conducted a mixed-method study with 24 UX evaluators using AI Assistant to identify UX problems from usability test videos
- Results show that providing explanations improves performance and engagement for UX evaluators, regardless of synchronous or asynchronous presentation
- Without explanations, synchronous AI improves performance and engagement more effectively than asynchronous AI
- Design implications presented for enhancing human-AI collaboration in AI-assisted UX evaluation
- Research contributes to understanding effective design of AI to assist UX evaluators in analyzing usability test videos, considering factors like explanations and synchronization.

## Summary

This study focuses on the design of AI systems to support effective collaboration between user experience (UX) evaluators and AI in the context of analyzing usability test videos. Two key design factors for AI-assisted UX evaluation are investigated: explanations and synchronization. Explanations involve AI informing humans about how it identifies UX problems during a usability test session, while synchronization refers to the ways in which humans and AI collaborate, either synchronously or asynchronously. To explore these design factors, a tool called AI Assistant was developed with four versions of user interfaces (UIs) representing different levels of explanations (with/without) and synchronization (sync/async). A hybrid wizard-of-oz approach was adopted to simulate an AI with reasonable performance. A mixed-method study was conducted with 24 UX evaluators using AI Assistant to identify UX problems from usability test videos. The results of the study demonstrate that providing explanations improves both performance and engagement for UX evaluators regardless of whether they are presented synchronously or asynchronously. However, when explanations are not provided, synchronous AI improves UX evaluators' performance and engagement more effectively compared to asynchronous AI. Based on their findings, design implications for enhancing human-AI collaboration in terms of AI-assisted UX evaluation were presented. This research contributes to understanding how AI can be designed to effectively assist UX evaluators in analyzing usability test videos by considering factors such as explanations and synchronization.