# Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness

**Authors:** Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, Shikun Zhang  
**arXiv:** [2304.11633](https://arxiv.org/abs/2304.11633)  
**Date:** 2023-04-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.11633v1/

## Key points

- Authors focus on assessing ChatGPT's ability to comprehend user intent and provide reasonable responses
- Evaluation conducted using 7 fine-grained information extraction (IE) tasks
- Systematic analysis of ChatGPT's performance in extracting relevant information from text
- Assessment of explainability, calibration, and faithfulness of ChatGPT's responses
- Comparison of evaluation results with inputs from ChatGPT itself and domain experts
- Poor performance in Standard-IE setting but excellent performance in OpenIE setting based on human evaluation
- High-quality and trustworthy explanations provided by ChatGPT for its decisions
- Issue of overconfidence leading to low calibration in predictions
- High level of faithfulness to the original text in most cases
- Manual annotation and release of test sets for the 7 fine-grained IE tasks comprising 14 datasets
- Datasets and code made available on GitHub for further research
- Insights provided into evaluating ChatGPT's information extraction capabilities
- Areas identified for potential improvements in performance and reliability.

## Summary

In the paper titled "Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness," authors Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang focus on assessing the overall ability of Large Language Models (LLMs) like ChatGPT in comprehending user intent and providing reasonable responses. They specifically evaluate ChatGPT using 7 fine-grained information extraction (IE) tasks. The authors present a systematic analysis by measuring ChatGPT's performance in terms of its ability to extract relevant information from text. They also assess its explainability, calibration and faithfulness to determine the reliability and accuracy of its responses. The evaluation results are compared with inputs from both ChatGPT itself and domain experts. The findings reveal that ChatGPT performs poorly in the Standard-IE setting but surprisingly exhibits excellent performance in the OpenIE setting based on human evaluation. Additionally, the research indicates that ChatGPT provides high-quality and trustworthy explanations for its decisions. However, there is an issue of overconfidence in predictions leading to low calibration. Furthermore, ChatGPT demonstrates a high level of faithfulness to the original text in most cases. To further promote research in this area, the authors manually annotate and release test sets for the 7 fine-grained IE tasks comprising 14 datasets. These datasets along with the code used in the study are made available on GitHub. Overall, this paper provides valuable insights into evaluating ChatGPT's information extraction capabilities and highlights areas where improvements can be made to enhance its performance and reliability.