# ChatGPT-3.5, ChatGPT-4, Google Bard, and Microsoft Bing to Improve Health Literacy and Communication in Pediatric Populations and Beyond

**Authors:** Kanhai S. Amin, Linda Mayes, Pavan Khosla, Rushabh Doshi  
**arXiv:** [2311.10075](https://arxiv.org/abs/2311.10075)  
**Date:** 2023-11-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.10075v1/

## Key points

- Enhanced health literacy linked to improved health outcomes
- Study explores using large language models (LLMs) to enhance health literacy in children and other populations
- Testing involved 288 conditions with 26 prompts using ChatGPT-3.5, Bing, and Google Bard; subset of 150 conditions further tested with ChatGPT-4 due to rate limits
- Primary outcome measurements focused on reading grade level (RGL) and word counts of LLM-generated output
- Basic prompts like "Explain" and "What is (are)" produced outputs at or above a 10th-grade RGL across all models
- LLMs varied in tailoring responses based on RGL when prompted to explain conditions from the 1st to 12th grade levels
- ChatGPT-3.5 and ChatGPT-4 performed better at achieving lower-grade level outputs compared to Bing and Bard
- Bard exhibited hesitancy in providing certain outputs, showing caution in health information dissemination
- Future research should verify accuracy and effectiveness of LLMs in improving health literacy
- Despite challenges in crafting outputs below a sixth-grade reading level, LLMs show potential in modifying higher-level outputs for enhanced communication among pediatric populations and beyond

## Summary

Enhanced health literacy has been associated with improved health outcomes. However, there is a lack of interventions studied in this area. This study explores the potential of large language models (LLMs) to enhance health literacy in children and other populations. A total of 288 conditions were tested using 26 different prompts through ChatGPT-3.5, Microsoft Bing, and Google Bard. Due to rate limits, a subset of 150 conditions was further tested through ChatGPT-4. The primary outcome measurements focused on the reading grade level (RGL) and word counts of the output generated by the LLMs. Results showed that basic prompts like "Explain" and "What is (are)" produced outputs at or above a 10th-grade RGL across all models. When prompts specified explaining conditions from the 1st to 12th RGL, LLMs demonstrated varying abilities to tailor responses based on RGL. ChatGPT-3.5 and ChatGPT-4 performed better in achieving lower-grade level outputs compared to Bing and Bard, which tended to consistently produce outputs at a high school level regardless of the prompt. Additionally, Bard exhibited hesitancy in providing certain outputs, indicating a cautious approach towards health information dissemination. While LLMs show promise in enhancing health communication, future research should focus on verifying their accuracy and effectiveness in improving health literacy. Despite challenges in crafting outputs below a sixth-grade reading level, LLMs have shown potential in modifying outputs above this threshold to enhance health literacy and communication among pediatric populations and beyond. The study compiled a comprehensive list of 288 childhood disorders from reputable sources like Johns Hopkins Children's Center and Seattle Children's Hospital for testing purposes. Prompt selection included simple queries as well as context-based prompts tailored for different grade levels to assess the LLMs' ability to communicate complex medical information effectively. To ensure standardized comparison, formatting was removed from outputs, ancillary information was excluded, and readability assessments were conducted using Gunning Fog and Flesch readability formulas. Overall, this study highlights the potential of LLMs in improving health literacy but underscores the need for further research to optimize their use in healthcare communication settings.