# Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models

**Authors:** Zhen Lin, Shubhendu Trivedi, Jimeng Sun  
**arXiv:** [2305.19187](https://arxiv.org/abs/2305.19187)  
**Date:** 2023-05-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.19187v1/

## Key points

- Authors Zhen Lin, Shubhendu Trivedi, and Jimeng Sun focus on uncertainty quantification for responses generated by large language models (LLMs) in natural language generation (NLG).
- Lack of research on uncertainty quantification for NLG with black-box access to LLMs due to closed-source nature and computational constraints.
- Differentiation between uncertainty and confidence in NLG tasks is crucial, with proposed metrics for measuring both.
- Simple yet effective metric based on average semantic dispersion identified as a reliable predictor of LLM response quality.
- Research offers valuable insights for practitioners managing uncertainty when utilizing LLMs for NLG tasks.

## Summary

, , , , 

In their paper titled "Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models," authors Zhen Lin, Shubhendu Trivedi, and Jimeng Sun address the challenges of assessing the trustworthiness of responses generated by large language models (LLMs) in natural language generation (NLG). While LLMs have shown promise in various domains, there is a lack of research on uncertainty quantification for NLG when dealing with black-box access to these models. The closed-source nature and computational constraints make it difficult to have white-box access. The study focuses on differentiating between uncertainty and confidence in NLG tasks, proposing and comparing metrics for measuring both. They find that a simple yet effective metric based on average semantic dispersion serves as a reliable predictor of LLM response quality. This research provides valuable insights for practitioners managing uncertainty when using LLMs for NLG tasks. By addressing challenges associated with black-box access, the study advances our understanding of gauging response reliability from LLMs. Code for replicating experiments is available at https://github.com/zlin7/UQ-NLG.