# Evaluating counterfactual explanations using Pearl's counterfactual method

**Authors:** Bevan I. Smith  
**arXiv:** [2301.02499](https://arxiv.org/abs/2301.02499)  
**Date:** 2023-01-06  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2301.02499v1/

## Key points

- Evaluation of counterfactual explanations (CEs) using Judea Pearl's method
- CEs are used to generate alternative scenarios for different outcomes
- Current CEs generated from machine learning models may lack consideration of true causal structure, leading to potential bias
- Research tested CEs using Pearl's method on three different causal structures to understand impact
- Thirty percent of CEs conflicted with those computed by Pearl's method, emphasizing the need for understanding true causal structure
- Importance of prioritizing causal discovery in generating CEs and modifying models accordingly
- Future work could involve conducting Pearl's causality model (PCM) using CEs on real-life data with known causal structures for validation
- Implementing CEs on real data and measuring outcomes over time is challenging but essential for gaining confidence in results
- Simulations involving hundreds of CEs applied to real-world data could provide valuable insights
- Understanding and incorporating true causal structures into CE generation processes is crucial for reliability and effectiveness

## Summary

In this study, the focus was on evaluating counterfactual explanations (CEs) using Judea Pearl's method of computing counterfactuals. CEs are methods used to generate alternative scenarios that lead to a different desirable outcome. For example, helping a student predicted to fail a course find ways to pass instead. However, CEs are currently generated from machine learning models that may not consider the true causal structure in the data. This can potentially introduce bias into the results. The research aimed to test CEs using Pearl's method and evaluate them on three different causal structures. The goal was to understand how the underlying causal structure impacts the generated CEs. Surprisingly, it was found that thirty percent of the CEs conflicted with those computed by Pearl's method. This highlights the importance of understanding the true causal structure before blindly relying on machine learning models for generating counterfactual explanations. Moving forward, it is crucial to prioritize causal discovery in generating CEs and modifying models accordingly. Future work could involve conducting Pearl's causality model (PCM) using CEs on real-life data with known causal structures for validation. While implementing CEs on real data and measuring outcomes over time is challenging, it remains an essential step towards gaining confidence in the results. To enhance confidence further, simulations involving hundreds of CEs and applying them to real-world data could provide valuable insights. Ultimately, understanding and incorporating true causal structures into CE generation processes will be key in ensuring their reliability and effectiveness in various applications within the data community.