# Whose Opinions Do Language Models Reflect?

**Authors:** Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, Tatsunori Hashimoto  
**arXiv:** [2303.17548](https://arxiv.org/abs/2303.17548)  
**Date:** 2023-03-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.17548v1/

## Key points

- Language models (LMs) in open-ended contexts and their impact on user satisfaction and societal views
- Proposal of a quantitative framework to investigate LM opinions using public opinion polls and human responses
- Creation of OpinionsQA dataset to evaluate alignment of LM opinions with US demographic groups across various topics
- Significant misalignment between current LMs and opinions of US demographic groups, comparable to Democrat-Republican divide on climate change
- Misalignment persists even when explicitly steering LMs towards specific demographic groups
- Left-leaning tendencies observed in some human feedback-tuned LMs
- Poor reflection of opinions from certain groups such as individuals aged 65+ and widowed individuals
- Code and data provided for further exploration at https://github.com/tatsu-lab/opinions_qa
- Importance of evaluating alignment of language models' opinions with diverse demographic perspectives.

## Summary

The authors address the increasing use of language models (LMs) in open-ended contexts and their potential impact on user satisfaction and societal views. To investigate the opinions reflected by LMs, they propose a quantitative framework leveraging high-quality public opinion polls and associated human responses. The authors create a new dataset called OpinionsQA to evaluate the alignment of LM opinions with those of 60 US demographic groups across various topics such as abortion and automation. The findings reveal significant misalignment between the views reflected by current LMs and those of US demographic groups comparable to the Democrat-Republican divide on climate change. Even when explicitly steering LMs towards specific demographic groups, this misalignment persists. This research confirms previous observations about left-leaning tendencies in some human feedback-tuned LMs but also highlights groups whose opinions are poorly reflected by current LMs such as individuals aged 65+ and widowed individuals. The authors provide their code and data for further exploration at https://github.com/tatsu-lab/opinions_qa which sheds light on potential biases present in language models' opinions and emphasizes the importance of evaluating their alignment with diverse demographic perspectives.