# Primacy Effect of ChatGPT

**Authors:** Yiwei Wang, Yujun Cai, Muhao Chen, Yuxuan Liang, Bryan Hooi  
**arXiv:** [2310.13206](https://arxiv.org/abs/2310.13206)  
**Date:** 2023-10-20  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2310.13206v1/

## Key points

- The paper explores if large language models like ChatGPT inherit human cognitive biases
- Experiments were conducted using ChatGPT to analyze its decision-making process based on label order in the prompt
- ChatGPT's decision is sensitive to label order and has a higher likelihood of selecting labels at earlier positions as the answer
- A metric called label order imbalance (LOI) was introduced to quantitatively evaluate this bias
- Results show that ChatGPT exhibits unfair treatment of label indices, particularly for relation label predictions and with increasing task difficulty
- Source code is released by the researchers for further exploration and development in this area
- This research contributes to understanding how large language models make decisions and highlights the need to address biases for more trustworthy outcomes.

## Summary

The paper titled "Primacy Effect of ChatGPT" explores whether large language models like ChatGPT inherit humans' cognitive biases. To investigate this, the authors conduct experiments using ChatGPT and analyze its decision-making process based on the order of labels in the prompt. They find that ChatGPT's decision is sensitive to the order of labels and has a higher likelihood of selecting labels at earlier positions as the answer. To evaluate this bias quantitatively, they introduce a metric called label order imbalance (LOI), which measures the disparity between predicted label indices and a uniform distribution. The results show that ChatGPT exhibits unfair treatment of label indices when making relation label predictions for input texts and that this unfairness increases with task difficulty. Furthermore, the researchers release their source code to facilitate further exploration and development in this area. Overall, this research contributes to our understanding of how large language models like ChatGPT make decisions and emphasizes the importance of addressing biases in these models for more trustworthy outcomes.