# Towards Measuring the Representation of Subjective Global Opinions in Language Models

**Authors:** Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, Deep Ganguli  
**arXiv:** [2306.16388](https://arxiv.org/abs/2306.16388)  
**Date:** 2023-06-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.16388v2/

## Key points

- Large Language Models (LLMs) may not accurately represent diverse global perspectives on societal issues
- Authors developed a quantitative framework to evaluate alignment of LLM-generated responses with human opinions
- GlobalOpinionQA dataset created from cross-national surveys captures wide range of opinions on global issues across countries
- Metric measures similarity between LLM-generated survey responses and human responses, considering country of origin
- Biases in default model responses tend to align more closely with populations from the USA, Europe, and South America
- Model's responses shift when prompted to consider a specific country's perspective but can reflect harmful cultural stereotypes inadvertently
- Translating GlobalOpinionQA questions into different languages does not always result in LLM responses aligning with speakers of those languages
- LLMs can perpetuate ideological assumptions and biases aligning with particular political viewpoints
- Importance of understanding how LLMs function in ambiguous settings to mitigate potential biases and build inclusive models respecting human diversity
- Authors aim to promote transparency in AI systems' values by releasing dataset for public use and providing interactive visualization tool
- Continued research needed to develop models that exhibit broad understanding of social contexts respectfully serving all individuals

## Summary

In the realm of Large Language Models (LLMs), there is a growing concern that these models may not accurately represent the diverse global perspectives on societal issues. To address this issue, the authors of this paper have developed a quantitative framework to evaluate whose opinions LLM-generated responses align with. They introduce the GlobalOpinionQA dataset, which consists of questions and answers from cross-national surveys aimed at capturing a wide range of opinions on global issues across different countries. The authors define a metric that measures the similarity between LLM-generated survey responses and human responses, taking into account the country of origin. Through three experiments conducted on an LLM trained to prioritize being helpful, honest, and harmless with Constitutional AI, they uncover biases in default model responses. These biases tend to align more closely with certain populations such as those from the USA, Europe, and South America. Furthermore, when prompted to consider a specific country's perspective, the model's responses shift accordingly but can inadvertently reflect harmful cultural stereotypes. Interestingly, translating GlobalOpinionQA questions into different languages does not necessarily result in LLM responses aligning with speakers of those languages. The study sheds light on how LLMs can perpetuate ideological assumptions and biases that align with particular political viewpoints. It also underscores the importance of understanding how these models function in settings involving ambiguity and nuance to mitigate potential biases and build more inclusive models that respect human diversity. By releasing their dataset for public use and providing an interactive visualization tool, the authors aim to promote transparency in AI systems' values and help researchers address social biases while striving towards developing models that are inclusive of diverse global viewpoints. The paper concludes by emphasizing the need for continued research into developing models that exhibit a broad understanding of social contexts to serve all individuals respectfully.