# PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering

**Authors:** Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, Weidi Xie  
**arXiv:** [2305.10415](https://arxiv.org/abs/2305.10415)  
**Date:** 2023-05-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.10415v1/

## Key points

- MedVInT: a generative model designed for Medical Visual Question Answering (MedVQA)
- PMC-VQA dataset: consists of 227k VQA pairs from 149k images covering various modalities and diseases
- Performance evaluation: pre-trained on PMC-VQA, fine-tuned on VQA-RAD and SLAKE benchmarks, outperforms existing methods significantly
- Importance of multimodal understanding: accurate answers depend on the relationship between images and questions posed
- Challenging nature of MedVQA dataset: even state-of-the-art models struggle, highlighting complexity and biomedical relevance

## Summary

, , , , 

This paper presents MedVInT, a generative model specifically designed for addressing the challenge of Medical Visual Question Answering (MedVQA). This task is crucial in efficiently interpreting medical images and extracting clinic-relevant information. The proposed approach reframes MedVQA as a generation task that involves human-machine interaction and utilizes a generative-based model to align visual information from a pre-trained vision encoder with a large language model. To evaluate the performance of this model, a scalable pipeline is established to construct the PMC-VQA dataset, which consists of 227k VQA pairs from 149k images covering various modalities and diseases. The proposed MedVInT model is pre-trained on PMC-VQA and fine-tuned on public benchmarks such as VQA-RAD and SLAKE, outperforming existing methods significantly. In addition, a challenging test set that underwent manual verification is introduced to further evaluate the performance of the model. Previous works in this field have utilized techniques such as Instruction Tuning with Large-language Models and Mini-GPT4 to improve performance by generating examples using ChatGPT. The field of MedVQA has gained interest recently; however, building robust systems remains challenging due to image complexity and limitations in available datasets. To address this issue, the authors introduce a new benchmark for MedVQA on PMC-VQA that evaluates different methods for both open-ended and multiple-choice tasks. Results demonstrate that multimodal understanding is crucial for accurate answers, highlighting the strong relationship between images and questions posed. Existing state-of-the-art multimodal models struggle on MedVQA tasks, showcasing the challenging nature of this dataset in terms of both its complexity and biomedical relevance. The PMC-VQA-test presents a significantly more challenging benchmark compared to previous models like PMC-CLIP. Even the best-performing models on natural images struggle with MedVQA questions, emphasizing the difficulty of this dataset as a robust benchmark for evaluating VQA models. Further comparisons of generative model backbones on PMC-VQA-test are discussed in detail. In summary, this paper introduces MedVInT, a generative model tailored for MedVQA tasks, along with constructing a comprehensive dataset (PMC-VQA) and providing state-of-the-art performance on existing benchmarks while setting a new standard for evaluating methods in this field. , , , , and  are the key concepts addressed in this paper.