Placing a human in the loop can help mitigate the risks of deploying AI systems in safety-critical settings. However, addressing the challenges arising from human error and uncertainty within human-AI interactions is crucial but understudied. In this study, the focus is on understanding human uncertainty in concept-based models, which are AI systems that allow human feedback through concept interventions. Previous research often assumes that humans are always certain and correct, but real-world decision-making involves occasional mistakes and uncertainty. To investigate how existing concept-based models handle uncertain interventions from humans, two novel datasets were used: UMNIST, a visual dataset with controlled simulated uncertainty based on the MNIST dataset; and CUB-S, a relabeling of the popular CUB concept dataset with rich soft labels provided by humans. The results show that training with uncertain concept labels can help address weaknesses in concept-based systems when dealing with uncertain interventions. However, there are challenges related to calibration errors among annotators. While most annotators are reasonably calibrated, some poorly calibrated annotations affect the overall calibration. Annotators tend to underestimate small probabilities and overestimate large probabilities. This behavior may be influenced by cognitive load reduction or rounding errors. These predictable errors can potentially be corrected when training an uncertainty-aware model. The poor calibration observed raises questions about whether it is due to the interface used or inherent limitations in eliciting uncertainties in a crowdsourcing setting where humans have limited cognitive resources at any given time. Regardless of the cause, it is crucial for systems to be robust to these nuances and peculiarities in elicited human uncertainty to ensure successful deployment. Applying computational investigations to CUB-S reveals that its richness poses a substantial challenge for concept-based models. While using models trained on coarse-grained uncertainty from CUB helps mitigate failures under test-time uncertainty to some extent, it is not a perfect solution. The study also explores how intervening with coarse-grained annotations impacts performance using bird species classification as the task. Evaluation protocols focus on task accuracy and Skyline interventions are included to demonstrate the best possible intervention policy and highlight the impact of different types of uncertainty on performance. Overall, this research highlights the importance of considering human uncertainty in AI systems and identifies open challenges that can be addressed through multidisciplinary research on building interactive uncertainty aware systems.
- Having a person involved in using AI systems in important situations can help make them safer.
- It is important to study and solve problems that happen when humans and AI work together.
- Sometimes people make mistakes or are unsure about things, which is different from what researchers thought before.
- Training AI systems with uncertain information can help them handle situations where things are not clear.
- There are challenges with getting accurate information from people who help annotate data for AI systems.
Definitions
- Mitigate: To make something less bad or harmful
- Safety-critical: Situations where safety is very important and mistakes could be dangerous
- Uncertainty: When you are not sure about something or don't have all the information
- Concept-based: Systems that use ideas or concepts to understand and make decisions
- Calibration errors: Mistakes made when trying to get accurate measurements or information
- Annotators: People who help add information or labels to data for AI systems
- Cognitive load reduction: Making things easier for your brain by reducing how much you have to think about
- Rounding errors: Mistakes made when rounding numbers up or down
Understanding Human Uncertainty in AI Systems: A Study on Concept-Based Models
In recent years, Artificial Intelligence (AI) systems have been increasingly deployed in safety-critical settings. To mitigate the risks associated with such deployments, it is important to understand how these systems interact with humans and address the challenges arising from human error and uncertainty. This study focuses on understanding human uncertainty in concept-based models, which are AI systems that allow for feedback through concept interventions.
Background
Previous research has often assumed that humans are always certain and correct when providing feedback to AI systems; however, real-world decision making involves occasional mistakes and uncertainty. To investigate how existing concept-based models handle uncertain interventions from humans, two novel datasets were used: UMNIST, a visual dataset with controlled simulated uncertainty based on the MNIST dataset; and CUB-S, a relabeling of the popular CUB concept dataset with rich soft labels provided by humans.
Results
The results show that training with uncertain concept labels can help address weaknesses in concept-based systems when dealing with uncertain interventions. However, there are challenges related to calibration errors among annotators. While most annotators are reasonably calibrated, some poorly calibrated annotations affect the overall calibration. Annotators tend to underestimate small probabilities and overestimate large probabilities; this behavior may be influenced by cognitive load reduction or rounding errors. These predictable errors can potentially be corrected when training an uncertainty-aware model. Applying computational investigations to CUB-S reveals that its richness poses a substantial challenge for concept-based models; while using models trained on coarse-grained uncertainty from CUB helps mitigate failures under test time uncertainty to some extent, it is not a perfect solution. The study also explores how intervening with coarse grained annotations impacts performance using bird species classification as the task; evaluation protocols focus on task accuracy and Skyline interventions are included to demonstrate the best possible intervention policy and highlight the impact of different types of uncertainty on performance.
Conclusion
Overall, this research highlights the importance of considering human uncertainty in AI systems and identifies open challenges that can be addressed through multidisciplinary research on building interactive uncertainty aware systems. Poor calibration observed raises questions about whether it is due to interface used or inherent limitations in eliciting uncertainties in crowdsourcing setting where humans have limited cognitive resources at any given time – regardless of cause it is crucial for system robustness against nuances & peculiarities in elicited human uncertainties for successful deployment
Placing a human in the loop may abate the risks of deploying AI systems in safety-critical settings (e.g., a clinician working with a medical AI system). However, mitigating risks arising from human error and uncertainty within such human-AI interactions is an important and understudied issue. In this work, we study human uncertainty in the context of concept-based models, a family of AI systems that enable human feedback via concept interventions where an expert intervenes on human-interpretable concepts relevant to the task. Prior work in this space often assumes that humans are oracles who are always certain and correct. Yet, real-world decision-making by humans is prone to occasional mistakes and uncertainty. We study how existing concept-based models deal with uncertain interventions from humans using two novel datasets: UMNIST, a visual dataset with controlled simulated uncertainty based on the MNIST dataset, and CUB-S, a relabeling of the popular CUB concept dataset with rich, densely-annotated soft labels from humans. We show that training with uncertain concept labels may help mitigate weaknesses of concept-based systems when handling uncertain interventions. These results allow us to identify several open challenges, which we argue can be tackled through future multidisciplinary research on building interactive uncertainty-aware systems. To facilitate further research, we release a new elicitation platform, UElic, to collect uncertain feedback from humans in collaborative prediction tasks.
