Results of the summarizing process for the arXiv paper: 2112.12387v1

This study focuses on the design of AI systems to support effective collaboration between user experience (UX) evaluators and AI in the context of analyzing usability test videos. Two key design factors for AI-assisted UX evaluation are investigated: explanations and synchronization. Explanations involve AI informing humans about how it identifies UX problems during a usability test session, while synchronization refers to the ways in which humans and AI collaborate, either synchronously or asynchronously. To explore these design factors, a tool called AI Assistant was developed with four versions of user interfaces (UIs) representing different levels of explanations (with/without) and synchronization (sync/async). A hybrid wizard-of-oz approach was adopted to simulate an AI with reasonable performance. A mixed-method study was conducted with 24 UX evaluators using AI Assistant to identify UX problems from usability test videos. The results of the study demonstrate that providing explanations improves both performance and engagement for UX evaluators regardless of whether they are presented synchronously or asynchronously. However, when explanations are not provided, synchronous AI improves UX evaluators' performance and engagement more effectively compared to asynchronous AI. Based on their findings, design implications for enhancing human-AI collaboration in terms of AI-assisted UX evaluation were presented. This research contributes to understanding how AI can be designed to effectively assist UX evaluators in analyzing usability test videos by considering factors such as explanations and synchronization.

• - Study focuses on designing AI systems for collaboration between UX evaluators and AI in analyzing usability test videos
• - Two key design factors investigated: explanations and synchronization
• - Explanations involve AI informing humans about how it identifies UX problems during usability tests
• - Synchronization refers to the ways humans and AI collaborate (synchronously or asynchronously)
• - Developed a tool called AI Assistant with four versions of user interfaces representing different levels of explanations and synchronization
• - Used a hybrid wizard-of-oz approach to simulate AI performance
• - Conducted a mixed-method study with 24 UX evaluators using AI Assistant to identify UX problems from usability test videos
• - Results show that providing explanations improves performance and engagement for UX evaluators, regardless of synchronous or asynchronous presentation
• - Without explanations, synchronous AI improves performance and engagement more effectively than asynchronous AI
• - Design implications presented for enhancing human-AI collaboration in AI-assisted UX evaluation
• - Research contributes to understanding effective design of AI to assist UX evaluators in analyzing usability test videos, considering factors like explanations and synchronization.

This study is about designing computer systems that can help people analyze videos to see if they are easy to use. They looked at two important things: explanations and synchronization. Explanations means the computer tells people how it finds problems in the videos. Synchronization means how well the computer and people work together, either at the same time or at different times. They made a tool called AI Assistant with different ways of explaining and working together. They tested it with 24 people who analyze videos, and found that giving explanations helps them do better, no matter if they work together at the same time or not. Without explanations, working together at the same time is better than working separately. This research helps us understand how to make computers help people analyze videos better." Definitions- AI: Artificial Intelligence - Computer systems that can think and learn like humans. - UX: User Experience - How easy or enjoyable something is for people to use. - Usability test: A test to see if something is easy for people to use. - Evaluators: People who look at something carefully to judge its quality or usefulness. - Collaboration: Working together as a team. - Performance: How well someone does their job or task. - Engagement: How interested and involved someone is in what they are doing. - Design implications: Ideas for how to make something better based on research findings.

Exploring Design Factors for AI-Assisted UX Evaluation

Usability testing is an important part of user experience (UX) evaluation, and it involves observing users as they interact with a product or service. As the number of usability tests increase, however, manually analyzing all the data can be time consuming and tedious. To address this challenge, researchers have proposed using artificial intelligence (AI) to support UX evaluators in their analysis process. This research paper focuses on exploring two key design factors for AI-assisted UX evaluation: explanations and synchronization.

Background

The use of AI to assist humans in various tasks has been studied extensively over the past few decades. In particular, there has been a growing interest in understanding how AI can be designed to effectively collaborate with humans in order to improve performance and engagement. In the context of UX evaluation, however, little is known about how best to design AI systems that support effective collaboration between UX evaluators and AI when analyzing usability test videos.

Methodology

To explore these design factors, a tool called AI Assistant was developed with four versions of user interfaces (UIs) representing different levels of explanations (with/without) and synchronization (sync/async). A hybrid wizard-of-oz approach was adopted to simulate an AI with reasonable performance. A mixed-method study was conducted with 24 UX evaluators using AI Assistant to identify UX problems from usability test videos.

Results & Findings

The results of the study demonstrate that providing explanations improves both performance and engagement for UX evaluators regardless of whether they are presented synchronously or asynchronously. However, when explanations are not provided, synchronous AI improves UX evaluators' performance and engagement more effectively compared to asynchronous AI.

Design Implications & Conclusions

Based on their findings, design implications for enhancing human-AI collaboration in terms of AI-assisted UX evaluation were presented. This research contributes to understanding how AI can be designed to effectively assist UX evaluators in analyzing usability test videos by considering factors such as explanations and synchronization