# EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Models

**Authors:** Abhay Gupta, Jacob Cheung, Philip Meng, Shayan Sayyed, Austen Liao, Kevin Zhu, Sean O'Brien  
**arXiv:** [2504.07100](https://arxiv.org/abs/2504.07100)  
**Date:** 2025-02-25  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2504.07100v1/

## Key points

- Authors address challenges of diversity in human language on NLP systems
- Existing benchmarks often overlook intra-language variations and neglect non-standard dialects
- Introduction of EnDive benchmark to assess LLM performance across various tasks
- Translation of datasets into underrepresented dialects using few-shot prompting techniques with input from native speakers
- Comparison against rule-based methods through fluency assessments, preference tests, and semantic similarity metrics
- Human evaluations validate high translation quality achieved by EnDive
- EnDive exposes significant disparities in performance between standard American English inputs and dialectal inputs across different large language models, highlighting inherent biases within these systems
- Research aims to promote inclusivity and fairness in language processing technologies by uncovering model biases and advocating for advancements

## Summary

In their paper titled "EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Models," authors Abhay Gupta, Jacob Cheung, Philip Meng, Shayan Sayyed, Austen Liao, Kevin Zhu, and Sean O'Brien address the challenges posed by the diversity of human language on natural language processing (NLP) systems. They highlight how existing benchmarks often fail to account for intra-language variations, thereby neglecting speakers of non-standard dialects. To bridge this gap, the authors introduce EnDive (English Diversity), a comprehensive benchmark that assesses the performance of five popular large language models (LLMs) across various tasks such as language understanding, algorithmic reasoning, mathematics, and logic. The framework developed by the authors involves translating datasets from Standard American English into five underrepresented dialects using few-shot prompting techniques with input from native speakers to ensure accuracy. These translations are then compared against rule-based methods through fluency assessments, preference tests, and semantic similarity metrics. Human evaluations validate the high translation quality achieved by EnDive, with average scores exceeding 6 out of 7 for faithfulness, fluency, and formality. By curating a challenging dataset that exposes significant disparities in performance between standard American English inputs and dialectal inputs across different large language models,

EnDive sheds light on inherent biases within these systems. The findings underscore the need for more equitable and dialect-aware NLP technologies. Through their research efforts,

the authors aim to promote inclusivity and fairness in language processing technologies by uncovering model biases and advocating for advancements in this field.