# Capabilities of GPT-4 on Medical Challenge Problems

**Authors:** Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, Eric Horvitz  
**arXiv:** [2303.13375](https://arxiv.org/abs/2303.13375)  
**Date:** 2023-03-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.13375v1/

## Key points

- Nori et al. evaluate the capabilities of GPT-4, a large language model (LLM), on medical competency examinations and benchmark datasets.
- Performance of GPT-4 is compared with earlier models, including GPT-3.5 and prompt-tuned versions of Flan-PaLM 540B.
- Analysis covers official practice materials for the United States Medical Licensing Examination (USMLE) and the MultiMedQA suite of benchmark datasets.
- GPT-4 surpasses the passing score on USMLE by over 20 points without any specialized prompt crafting.
- GPT-4 outperforms earlier general-purpose models (GPT-3.5) as well as prompt-tuned models specifically fine-tuned on medical knowledge (Med-PaLM).
- GPT-4 demonstrates significantly improved probability calibration compared to GPT-3.5, indicating enhanced ability to predict answer correctness.
- Qualitative exploration shows that GPT-4 can explain medical reasoning, personalize explanations to students, and create new counterfactual scenarios around a medical case.
- Implications for potential uses of GPT-4 in medical education, assessment, and clinical practice are discussed.
- Challenges related to accuracy and safety when deploying such models in real-world healthcare settings are emphasized.

## Summary

In their study, Nori et al. evaluate the capabilities of GPT-4, a large language model (LLM), on medical competency examinations and benchmark datasets. They compare the performance of GPT-4 with earlier models, including GPT-3.5 and prompt-tuned versions of Flan-PaLM 540B. The analysis covers official practice materials for the United States Medical Licensing Examination (USMLE) and the MultiMedQA suite of benchmark datasets. The researchers conduct experiments to investigate various aspects of GPT-4's performance such as its ability to handle test questions containing both text and images, its memorization during training, and its probability calibration which is crucial in high-stakes applications like medicine. The results show that GPT-4 surpasses the passing score on USMLE by over 20 points without any specialized prompt crafting. It also outperforms earlier general-purpose models (GPT-3.5) as well as prompt-tuned models specifically fine-tuned on medical knowledge (Med-PaLM). Additionally, GPT-4 demonstrates significantly improved probability calibration compared to GPT-3.5, indicating its enhanced ability to predict answer correctness. The researchers also qualitatively explore the behavior of GPT-4 through a case study. They demonstrate its capability to explain medical reasoning, personalize explanations to students, and interactively create new counterfactual scenarios around a medical case. The findings have implications for potential uses of GPT-4 in medical education, assessment, and clinical practice; however they emphasize the importance of addressing challenges related to accuracy and safety when deploying such models in real world healthcare settings. Overall, this comprehensive evaluation highlights the remarkable capabilities of GPT-4 in natural language understanding and generation within the domain of medicine.