# PMC-LLaMA: Further Finetuning LLaMA on Medical Papers

**Authors:** Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie  
**arXiv:** [2304.14454](https://arxiv.org/abs/2304.14454)  
**Date:** 2023-04-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.14454v1/

## Key points

- Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding across various domains
- In areas that require precision, such as medical applications, these models often exhibit unsatisfactory performance due to a lack of domain-specific knowledge
- PMC-LLaMA is an open-source language model that is fine-tuned on 4.8 million biomedical academic papers to inject medical knowledge and enhance its capability in the medical domain
- The authors conducted a preliminary investigation by fine-tuning the existing LLaMA model with the aforementioned dataset and demonstrated that PMC-LLaMA is more suitable for medical tasks compared to LLaMA
- The evaluation was conducted on three biomedical QA datasets: PubMedQA, MedMCQA, and USMLE, showing better understanding of biomedical domain-specific concepts and achieving high performance on QA benchmarks after fine-tuning
- The authors outline their fine-tuning procedure using S2ORC datasets with specific training details such as max context length set at 512, batch size at 128, AdamW optimizer with learning rate 2e-5 and Fully Sharded Data Parallel (FSDP) acceleration strategy and bf16 (Brain Floating Point) data format.
- The model is trained for five epochs with eight A100 GPUs in around seven days, and in each epoch they randomly sample 512 continuous tokens per paper for training.
- The authors also provide a detailed description of the evaluation benchmark which includes three QA datasets: PubMedQA, MedMCQA and UMLSE.
- Overall , PMC - LLaMA offers an open - source language model that enhances LLaMA's capability in the medical domain by injecting domain - specific knowledge through fine - tuning on biomedical academic papers .
- -The model and codes are publicly available along with an online demo for further exploration.

## Summary

Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding across various domains. However, in areas that require precision, such as medical applications, these models often exhibit unsatisfactory performance due to a lack of domain-specific knowledge. In this report, the authors introduce PMC-LLaMA, an open-source language model that is fine-tuned on 4.8 million biomedical academic papers to inject medical knowledge and enhance its capability in the medical domain. The authors conduct a preliminary investigation by fine-tuning the existing LLaMA model with the aforementioned dataset and demonstrate that PMC-LLaMA is more suitable for medical tasks compared to LLaMA. The evaluation is conducted on three biomedical QA datasets: PubMedQA, MedMCQA, and USMLE. The results show that after fine-tuning, PMC-LLaMA demonstrates better understanding of biomedical domain-specific concepts and achieves high performance on QA benchmarks. In terms of experiment details, the authors outline their fine-tuning procedure using S2ORC datasets with 81.1M English-language academic papers filtered with PubMed Central (PMC)-id resulting in around 4.9M highly related papers totaling over 75B tokens. They fine-tune the LLaMA-7B model on these open-accessed PMC papers using an autoregressive generation objective introduced in GPT2 with specific training details such as max context length set at 512, batch size at 128, AdamW optimizer with learning rate 2e-5 and Fully Sharded Data Parallel (FSDP) acceleration strategy and bf16 (Brain Floating Point) data format. The model is trained for five epochs with eight A100 GPUs in around seven days, and in each epoch they randomly sample 512 continuous tokens per paper for training. The authors also provide a detailed description of the evaluation benchmark which includes three QA datasets: PubMedQA, MedMCQA and UMLSE. PubMedQA contains questions on biomedical research where the model is provided with paper abstracts from PubMed and required to complete multiple choice questions while MedMCQA is a dataset of multiple choice questions sourced from mock exams and past exams of two Indian medical school entrance exams called AIIMS and NEET PG each question having four choices . Overall , PMC - LLaMA offers an open - source language model that enhances LLaMA's capability in the medical domain by injecting domain - specific knowledge through fine - tuning on biomedical academic papers . The model and codes are publicly available along with an online demo for further exploration .