# PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology

**Authors:** Yuxuan Sun, Hao Wu, Chenglu Zhu, Sunyi Zheng, Qizi Chen, Kai Zhang, Yunlong Zhang, Xiaoxiao Lan, Mengyue Zheng, Jingxiong Li, Xinheng Lyu, Tao Lin, Lin Yang  
**arXiv:** [2401.16355](https://arxiv.org/abs/2401.16355)  
**Date:** 2024-01-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.16355v1/

## Key points

- The introduction of large multimodal models has revolutionized advancements in pathology.
- Lack of specialized, high-quality benchmarks has hindered the development and evaluation of these models.
- PathMMU is a groundbreaking initiative that serves as the largest and most meticulously vetted pathology benchmark.
- PathMMU comprises 33,573 multimodal multi-choice questions and 21,599 images with detailed explanations for each question.
- It leverages GPT-4V to generate an extensive set of Questions and Answers (Q&As).
- A panel of six esteemed pathologists rigorously scrutinizes each question within PathMMU's validation and test sets to establish an expert-level performance benchmark.
- Empirical evaluations show that even advanced LMMs struggle with the challenges posed by PathMMU.
- The top-performing LLM, GPT-4V, achieves a modest zero-shot performance rate of 51.7%, lower than human pathologists' 71.4%.
- Certain open-sourced LMMs surpass GPT-4V after fine-tuning but still fall short compared to human pathologists' expertise.
- PathMMU aims to provide invaluable insights and catalyze advancements in developing more specialized next-generation Large Multimodal Models for pathology applications.

## Summary

In the realm of Artificial Intelligence, the advent of large multimodal models has revolutionized the potential for advancements in pathology. However, a significant hindrance to their development and accurate evaluation has been the absence of specialized, high-quality benchmarks. To tackle this challenge head-on, a groundbreaking initiative known as PathMMU has been introduced. stands as the largest and most meticulously vetted pathology benchmark tailored specifically for . This comprehensive benchmark comprises an impressive array of 33,573 multimodal multi-choice questions accompanied by 21,599 images sourced from various channels. Notably, each question is complemented by a detailed explanation elucidating the correct answer. The creation of  leverages the robust capabilities of GPT-4V, harnessing approximately 30,000 curated image-caption pairs to generate an extensive set of Questions and Answers (Q&As). To ensure the utmost credibility and reliability of , a panel of six esteemed pathologists rigorously scrutinize each question within both its validation and test sets. Simultaneously, these experts establish an expert-level performance benchmark against which  can be measured. A series of exhaustive evaluations have been conducted to gauge the efficacy and resilience of 14 open-sourced and three closed-sourced LMMs when faced with the formidable challenges posed by . Additionally, representative LMMs have undergone fine-tuning processes to assess their adaptability to this rigorous benchmark. The empirical findings reveal that even advanced LMMs struggle when confronted with the complexities inherent in the  benchmark. The top-performing LLM, GPT-4V, achieves a modest zero-shot performance rate of only 51.7%, significantly lower than the impressive 71.4% demonstrated by human pathologists. Following fine-tuning efforts, certain open-sourced LMMs manage to surpass GPT-4V with performance rates exceeding 60%. However, they still fall short when compared to the expertise exhibited by human pathologists. It is hoped that through its establishment and meticulous curation process,  will not only provide invaluable insights but also catalyze advancements in developing more specialized next-generation Large Multimodal Models tailored specifically for applications in pathology.