This study focuses on the design of AI systems to support effective collaboration between user experience (UX) evaluators and AI in the context of analyzing usability test videos. Two key design factors for AI-assisted UX evaluation are investigated: explanations and synchronization. Explanations involve AI informing humans about how it identifies UX problems during a usability test session, while synchronization refers to the ways in which humans and AI collaborate, either synchronously or asynchronously. To explore these design factors, a tool called AI Assistant was developed with four versions of user interfaces (UIs) representing different levels of explanations (with/without) and synchronization (sync/async). A hybrid wizard-of-oz approach was adopted to simulate an AI with reasonable performance. A mixed-method study was conducted with 24 UX evaluators using AI Assistant to identify UX problems from usability test videos. The results of the study demonstrate that providing explanations improves both performance and engagement for UX evaluators regardless of whether they are presented synchronously or asynchronously. However, when explanations are not provided, synchronous AI improves UX evaluators' performance and engagement more effectively compared to asynchronous AI. Based on their findings, design implications for enhancing human-AI collaboration in terms of AI-assisted UX evaluation were presented. This research contributes to understanding how AI can be designed to effectively assist UX evaluators in analyzing usability test videos by considering factors such as explanations and synchronization.
This study is about designing computer systems that can help people analyze videos to see if they are easy to use. They looked at two important things: explanations and synchronization. Explanations means the computer tells people how it finds problems in the videos. Synchronization means how well the computer and people work together, either at the same time or at different times. They made a tool called AI Assistant with different ways of explaining and working together. They tested it with 24 people who analyze videos, and found that giving explanations helps them do better, no matter if they work together at the same time or not. Without explanations, working together at the same time is better than working separately. This research helps us understand how to make computers help people analyze videos better."
Definitions
- AI: Artificial Intelligence - Computer systems that can think and learn like humans.
- UX: User Experience - How easy or enjoyable something is for people to use.
- Usability test: A test to see if something is easy for people to use.
- Evaluators: People who look at something carefully to judge its quality or usefulness.
- Collaboration: Working together as a team.
- Performance: How well someone does their job or task.
- Engagement: How interested and involved someone is in what they are doing.
- Design implications: Ideas for how to make something better based on research findings.
Exploring Design Factors for AI-Assisted UX Evaluation
Usability testing is an important part of user experience (UX) evaluation, and it involves observing users as they interact with a product or service. As the number of usability tests increase, however, manually analyzing all the data can be time consuming and tedious. To address this challenge, researchers have proposed using artificial intelligence (AI) to support UX evaluators in their analysis process. This research paper focuses on exploring two key design factors for AI-assisted UX evaluation: explanations and synchronization.
Background
The use of AI to assist humans in various tasks has been studied extensively over the past few decades. In particular, there has been a growing interest in understanding how AI can be designed to effectively collaborate with humans in order to improve performance and engagement. In the context of UX evaluation, however, little is known about how best to design AI systems that support effective collaboration between UX evaluators and AI when analyzing usability test videos.
Methodology
To explore these design factors, a tool called AI Assistant was developed with four versions of user interfaces (UIs) representing different levels of explanations (with/without) and synchronization (sync/async). A hybrid wizard-of-oz approach was adopted to simulate an AI with reasonable performance. A mixed-method study was conducted with 24 UX evaluators using AI Assistant to identify UX problems from usability test videos.
Results & Findings
The results of the study demonstrate that providing explanations improves both performance and engagement for UX evaluators regardless of whether they are presented synchronously or asynchronously. However, when explanations are not provided, synchronous AI improves UX evaluators' performance and engagement more effectively compared to asynchronous AI.
Design Implications & Conclusions
Based on their findings, design implications for enhancing human-AI collaboration in terms of AI-assisted UX evaluation were presented. This research contributes to understanding how AI can be designed to effectively assist UX evaluators in analyzing usability test videos by considering factors such as explanations and synchronization
Analyzing usability test videos is arduous. Although recent research showed the promise of AI in assisting with such tasks, it remains largely unknown how AI should be designed to facilitate effective collaboration between user experience (UX) evaluators and AI. Inspired by the concepts of agency and work context in human and AI collaboration literature, we studied two corresponding design factors for AI-assisted UX evaluation: explanations and synchronization. Explanations allow AI to further inform humans how it identifies UX problems from a usability test session; synchronization refers to the two ways humans and AI collaborate: synchronously and asynchronously. We iteratively designed a tool, AI Assistant, with four versions of UIs corresponding to the two levels of explanations (with/without) and synchronization (sync/async). By adopting a hybrid wizard-of-oz approach to simulating an AI with reasonable performance, we conducted a mixed-method study with 24 UX evaluators identifying UX problems from usability test videos using AI Assistant. Our quantitative and qualitative results show that AI with explanations, regardless of being presented synchronously or asynchronously, provided better support for UX evaluators' analysis and was perceived more positively; when without explanations, synchronous AI better improved UX evaluators' performance and engagement compared to the asynchronous AI. Lastly, we present the design implications for AI-assisted UX evaluation and facilitating more effective human-AI collaboration.
