# Large Language Models are Geographically Biased

**Authors:** Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, Stefano Ermon  
**arXiv:** [2402.02680](https://arxiv.org/abs/2402.02680)  
**Date:** 2024-02-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.02680v1/

## Key points

- Large Language Models (LLMs) carry inherent biases from their training data, perpetuating societal harm
- Understanding and evaluating biases in LLMs is crucial for fairness and accuracy as they grow in influence
- Study by Stanford University researchers focused on examining geographic biases in LLMs
- Geospatial predictions made by LLMs revealed systemic errors defined as problematic geographic biases
- LLMs accurately made zero-shot geospatial predictions but exhibited biases against locations with lower socioeconomic conditions, especially in regions like Africa
- Bias manifested in subjective topics such as attractiveness, morality, and intelligence with significant variation among different existing LLMs
- Importance of addressing geographic biases in LLMs to ensure fairness and accuracy across various fields

## Summary

Large Language Models (LLMs) have been shown to carry inherent biases from their training data, which can perpetuate societal harm. As these foundational models continue to grow in influence across various domains, it becomes increasingly important to understand and evaluate their biases for the sake of fairness and accuracy. In a recent study by Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, and Stefano Ermon from Stanford University, the focus was on examining the geographic biases present in LLMs. The researchers proposed studying what LLMs understand about our world through a geographical lens. This approach is powerful because geographic space serves as a tangible representation of various aspects of human life such as culture, race, language, politics, and religion. By analyzing geospatial predictions made by LLMs, the researchers identified systemic errors in these predictions that they defined as problematic geographic biases. Initially, the study demonstrated that LLMs are capable of accurately making zero-shot geospatial predictions that show strong correlation with ground truth data. These predictions exhibited a Spearman's ρ value of up to 0.89, indicating high accuracy. However, further analysis revealed common biases across objective and subjective topics within these predictions. One significant finding was that LLMs displayed bias against locations with lower socioeconomic conditions, particularly in regions like Africa. This bias manifested in subjective topics such as attractiveness, morality, and intelligence where Spearman's ρ values reached up to 0.70. To quantify this bias more systematically, the researchers introduced a bias score and found significant variation in the magnitude of bias among different existing LLMs. Overall,this study sheds light on the presence of geographic biases in Large Language Models and underscores the importance of addressing these biases to ensure fairness and accuracy in their applications across various fields. The findings highlight the need for ongoing research and development efforts aimed at mitigating biases in LLMs for more equitable outcomes in society.