# A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models

**Authors:** Stephen R. Pfohl, Heather Cole-Lewis, Rory Sayres, Darlene Neal, Mercy Asiedu, Awa Dieng, Nenad Tomasev, Qazi Mamunur Rashid, Shekoofeh Azizi, Negar Rostamzadeh, Liam G. McCoy, Leo Anthony Celi, Yun Liu, Mike Schaekermann, Alanna Walton, Alicia Parrish, Chirag Nagpal, Preeti Singh, Akeiylah Dewitt, Philip Mansfield  
**arXiv:** [2403.12025](https://arxiv.org/abs/2403.12025)  
**Date:** 2024-03-18  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2403.12025v1/

## Key points

- Large language models (LLMs) have the potential to revolutionize healthcare by providing complex information and answering medical questions.
- LLMs pose risks of introducing biases that can exacerbate health disparities.
- A team of researchers developed resources and methodologies to identify biases in long-form LLM-generated answers related to health equity.
- They conducted an empirical case study using Med-PaLM 2, resulting in the largest human evaluation study in this field to date.
- The team introduced a multifactorial framework for assessing LLM-generated answers for biases and created EquityMedQA, a collection of seven datasets containing both manually curated and LLM-generated questions enriched with adversarial queries.
- Utilizing diverse datasets curated through various methods helped uncover biases that might have been overlooked with narrower evaluation approaches.
- The team emphasized the importance of employing diverse assessment methodologies and engaging raters from different backgrounds and expertise levels.
- While their framework could pinpoint specific forms of bias, they acknowledged its limitations in providing a holistic assessment of whether deploying an AI system promotes equitable health outcomes.
- The broader community is encouraged to leverage and expand upon their tools and methods towards achieving a common goal of LLMs that support accessible and equitable healthcare for all.

## Summary

Large language models (LLMs) have the potential to revolutionize healthcare by providing complex information and answering medical questions. However, they also pose risks of introducing biases that can exacerbate health disparities. To address this issue, a team of researchers developed resources and methodologies to identify biases in long-form LLM-generated answers related to health equity. They conducted an empirical case study using Med-PaLM 2, resulting in the largest human evaluation study in this field to date. The  introduced a multifactorial framework for assessing LLM-generated answers for biases and created EquityMedQA, a collection of seven datasets containing both manually curated and LLM-generated questions enriched with adversarial queries. Their approach was grounded in an iterative participatory process that reviewed potential biases in Med-PaLM 2 responses to adversarial queries. Through their study, the  found that utilizing diverse datasets curated through various methods, along with a comprehensive evaluation protocol involving multiple assessment rubric designs and diverse rater groups, helped uncover biases that might have been overlooked with narrower evaluation approaches. They emphasized the importance of employing diverse assessment methodologies and engaging raters from different backgrounds and expertise levels. While their framework could pinpoint specific forms of bias, the  acknowledged its limitations in providing a holistic assessment of whether deploying an AI system promotes equitable health outcomes. They encouraged the broader community to leverage and expand upon their tools and methods towards achieving a common goal of LLMs that support accessible and equitable healthcare for all. The team included a diverse group of authors who contributed to this important work aimed at addressing bias in large language models within the healthcare domain.