# Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation

**Authors:** Jinglong Gao, Xiao Ding, Bing Qin, Ting Liu  
**arXiv:** [2305.07375](https://arxiv.org/abs/2305.07375)  
**Date:** 2023-05-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.07375v3/

## Key points

- The paper evaluates ChatGPT's causal reasoning capabilities, which are important for NLP applications.
- Despite performing well in various NLP tasks, it is unclear how well ChatGPT performs in causal reasoning.
- Experiments were conducted using four versions of ChatGPT and the Event Causality Identification (ECI) task as a benchmark.
- Results show that ChatGPT is a good causal interpreter but not a good causal reasoner due to reporting biases and upgrading processes such as RLHF.
- In-Context Learning (ICL) and Chain-of-Thought (COT) techniques can exacerbate ChatGPT's causal hallucination.
- The ability of ChatGPT to reason causally is sensitive to the words used to express the causal concept in prompts, with close-ended prompts performing better than open-ended ones.
- ChatGPT excels at capturing explicit causality rather than implicit causality and performs better in sentences with lower event density and smaller lexical distance between events.
- F1 score was used as an evaluation metric for the experiments.
- This study provides insights into the limitations of current language models for understanding causality in natural language text.

## Summary

This paper presents a comprehensive evaluation of the causal reasoning capabilities of ChatGPT, which is crucial for numerous NLP applications. Despite its impressive performance in various NLP tasks, it is unclear how well ChatGPT performs in causal reasoning. The authors conduct experiments using four state-of-the-art versions of ChatGPT and utilize the Event Causality Identification (ECI) task as a comprehensive causal reasoning benchmark. The results show that ChatGPT is not a good causal reasoner but rather a good causal interpreter. Additionally, ChatGPT has a serious hallucination on causal reasoning due to reporting biases between causal and non-causal relationships in natural language and upgrading processes such as RLHF. The In-Context Learning (ICL) and Chain-of-Thought (COT) techniques can further exacerbate such causal hallucination. Furthermore, the authors find that the causal reasoning ability of ChatGPT is sensitive to the words used to express the causal concept in prompts, and close-ended prompts perform better than open-ended prompts. For events in sentences, ChatGPT excels at capturing explicit causality rather than implicit causality and performs better in sentences with lower event density and smaller lexical distance between events. Finally, the authors use F1 score as an evaluation metric for their experiments. Overall, this study provides important insights into the limitations of current language models for understanding causality in natural language text.