# Human Uncertainty in Concept-Based AI Systems

**Authors:** Katherine M. Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, Krishnamurthy Dvijotham  
**arXiv:** [2303.12872](https://arxiv.org/abs/2303.12872)  
**Date:** 2023-03-22  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2303.12872v1/

## Key points

- Placing a human in the loop can mitigate risks in deploying AI systems in safety-critical settings.
- Addressing challenges arising from human error and uncertainty within human-AI interactions is crucial but understudied.
- Real-world decision-making involves occasional mistakes and uncertainty, contrary to assumptions made in previous research.
- Training with uncertain concept labels can help address weaknesses in concept-based systems when dealing with uncertain interventions.
- Challenges related to calibration errors among annotators exist, with some poorly calibrated annotations affecting overall calibration.
- Annotators tend to underestimate small probabilities and overestimate large probabilities, potentially influenced by cognitive load reduction or rounding errors.
- Poor calibration raises questions about the interface used or inherent limitations in eliciting uncertainties in a crowdsourcing setting with limited cognitive resources.
- The richness of the CUB-S dataset poses a substantial challenge for concept-based models.
- Using models trained on coarse-grained uncertainty from CUB helps mitigate failures under test-time uncertainty to some extent but is not perfect.
- Intervening with coarse-grained annotations impacts performance in bird species classification task.
- Evaluation protocols focus on task accuracy and Skyline interventions demonstrate the best possible intervention policy and highlight impact of different types of uncertainty on performance.

## Summary

Placing a human in the loop can help mitigate the risks of deploying AI systems in safety-critical settings. However, addressing the challenges arising from human error and uncertainty within human-AI interactions is crucial but understudied. In this study, the focus is on understanding human uncertainty in concept-based models, which are AI systems that allow human feedback through concept interventions. Previous research often assumes that humans are always certain and correct, but real-world decision-making involves occasional mistakes and uncertainty. To investigate how existing concept-based models handle uncertain interventions from humans, two novel datasets were used: UMNIST, a visual dataset with controlled simulated uncertainty based on the MNIST dataset; and CUB-S, a relabeling of the popular CUB concept dataset with rich soft labels provided by humans. The results show that training with uncertain concept labels can help address weaknesses in concept-based systems when dealing with uncertain interventions. However, there are challenges related to calibration errors among annotators. While most annotators are reasonably calibrated, some poorly calibrated annotations affect the overall calibration. Annotators tend to underestimate small probabilities and overestimate large probabilities. This behavior may be influenced by cognitive load reduction or rounding errors. These predictable errors can potentially be corrected when training an uncertainty-aware model. The poor calibration observed raises questions about whether it is due to the interface used or inherent limitations in eliciting uncertainties in a crowdsourcing setting where humans have limited cognitive resources at any given time. Regardless of the cause, it is crucial for systems to be robust to these nuances and peculiarities in elicited human uncertainty to ensure successful deployment. Applying computational investigations to CUB-S reveals that its richness poses a substantial challenge for concept-based models. While using models trained on coarse-grained uncertainty from CUB helps mitigate failures under test-time uncertainty to some extent, it is not a perfect solution. The study also explores how intervening with coarse-grained annotations impacts performance using bird species classification as the task. Evaluation protocols focus on task accuracy and Skyline interventions are included to demonstrate the best possible intervention policy and highlight the impact of different types of uncertainty on performance. Overall, this research highlights the importance of considering human uncertainty in AI systems and identifies open challenges that can be addressed through multidisciplinary research on building interactive uncertainty aware systems.