# Dynamic Competency Self-Assessment for Autonomous Agents

**Authors:** Nicholas Conlon, Nisar R. Ahmed, Daniel Szafir  
**arXiv:** [2303.01646](https://arxiv.org/abs/2303.01646)  
**Date:** 2023-03-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.01646v1/

## Key points

- Deployment of autonomous robots in complex environments presents challenges such as platform degradation, environmental uncertainties, and deviations from validated operation conditions.
- These factors can make it difficult for human partners to understand the capabilities and limitations of robots.
- Self-assessment of robot competency in dynamic and uncertain environments is crucial for successful human-robot teaming.
- An Event-Triggered Generalized Outcome Assessment (ET-GOA) algorithm was evaluated for autonomous agents to dynamically assess task confidence during execution.
- The ET-GOA algorithm uses a fast online statistical test of the agent's observations and its model predictions to determine when competency assessment is necessary.
- Experimental results using ET-GOA to generate competency reports during a simulated delivery task showed significant effects on deliveries in both static and dynamic environments.
- In the static environment, there were significant increases in deliveries with ET-GOA compared to random and with GOA compared to random. There was no difference between GOA and ET-GOA in the static environment.
- In the dynamic environment, there were significant differences between all reporting conditions on deliveries.
- Self-assessing agents using algorithms such as ET-GOA can improve their performance in delivering tasks by dynamically assessing their own competency levels during execution.

## Summary

The deployment of autonomous robots in complex environments presents challenges such as platform degradation, environmental uncertainties, and deviations from validated operation conditions. These factors can make it difficult for human partners to understand the capabilities and limitations of robots. To address this issue, self-assessment of robot competency in dynamic and uncertain environments is crucial for successful human-robot teaming. In this study presented at the HRI 2023 workshop on Variable Autonomy for Human-Robot Teaming, an Event-Triggered Generalized Outcome Assessment (ET-GOA) algorithm was evaluated for autonomous agents to dynamically assess task confidence during execution. The ET-GOA algorithm uses a fast online statistical test of the agent's observations and its model predictions to determine when competency assessment is necessary. The agent then selects a goal based on equation 2 and navigates directly to that goal until it either reaches the goal or triggers a re-assessment and chooses a new goal. Experimental results using ET-GOA to generate competency reports during a simulated delivery task showed significant effects of each reporting condition on deliveries in both static and dynamic environments. In the static environment, there were significant increases in deliveries with ET-GOA compared to random and with GOA compared to random. There was no difference between GOA and ET-GOA in the static environment. In contrast, in the dynamic environment, there were significant differences between all reporting conditions on deliveries. Overall, this study demonstrates that self-assessing agents using algorithms such as ET-GOA can improve their performance in delivering tasks by dynamically assessing their own competency levels during execution. Future research directions include exploring other types of statistical tests for competency assessment and investigating how these assessments can be used to improve human-robot teaming further.