Results of the summarizing process for the arXiv paper: 2311.16079v1

In this work, the researchers introduce MEDITRON, a suite of open-source large language models (LLMs) specifically designed for the medical domain. These LLMs have 7B and 70B parameters and are built upon Llama-2 using Nvidia's Megatron-LM distributed trainer. To train these models, a comprehensively curated medical corpus was used, which includes selected PubMed articles, abstracts, and internationally-recognized medical guidelines. The GUIDELINES corpus included in the training data covers a wide range of contexts in terms of geographic scope (from global to institutional), resource settings (high-, low-, and volatile-resource settings), and target audiences (clinicians or patients). It also incorporates various peer review processes ranging from UN bodies to publicly crowdsourced knowledge bases. To create the pre-training mix for MEDITRON, PubMed papers and abstracts were chosen due to their vast amount of biomedical textual data. The researchers collected 4.47M full-text papers from the PubMed Central Open Access Subset as well as 444,521 open-access full-text PubMed papers that were not found in the archive. Additionally, 16,209,047 PubMed and PubMed Central abstracts were collected with a knowledge cutoff date of August 2023. To preprocess the content from these sources metadata information references acknowledgments tables figures URLs and non-English content were removed while inline citations section headers figures tables mathematical formulas were identified using special tokens. In-text references were formatted with a similar methodology to the Galactica model to promote accurate citations by replacing them with [BIB_REF] tokens containing truncated titles and authors' last names while figure and table references were wrapped with [FIG_REF] tokens containing figure numbers and truncated captions. Section headers were indicated with '#' and '##' tokens for main sections and subsections respectively. Experience replay was employed in the training process to overcome catastrophic forgetting which helps prevent the model from forgetting previously learned information by including data from old tasks when training on new tasks.

• MEDITRON is a suite of open-source large language models (LLMs) designed for the medical domain.
• The LLMs have 7B and 70B parameters and are built upon Llama-2 using Nvidia's Megatron-LM distributed trainer.
• A comprehensively curated medical corpus, including PubMed articles, abstracts, and internationally recognized medical guidelines, was used to train these models.
• The training data covers a wide range of contexts in terms of geographic scope, resource settings, and target audiences.
• PubMed papers and abstracts were chosen to create the pre-training mix for MEDITRON due to their vast amount of biomedical textual data.
• Metadata information, non-English content, acknowledgments tables figures URLs were removed during preprocessing.
• In-text references were formatted using special tokens for accurate citations.
• Section headers were indicated with specific tokens for main sections and subsections.
• Experience replay was employed in the training process to prevent forgetting previously learned information.

MEDITRON is a special computer program that helps doctors with medical information. It has been made using a lot of words and rules to understand the medical field better. The program was trained using many different sources like medical articles and guidelines. They took out some extra information from these sources to make it easier to read. They also used a special method called experience replay to remember things they learned before." Definitions- Open-source: A type of computer program that anyone can use and change for free. - Large language models (LLMs): Special computer programs that understand and generate human language. - Parameters: The settings or instructions that tell a computer program how to work. - Corpus: A collection of written or spoken texts used for studying or training a computer program. - Pre-training mix: The combination of different texts used to teach the MEDITRON program. - Metadata: Information about other information, like details about where an article comes from. - In-text references: Special markers in an article that show where certain information comes from. - Section headers: Titles or labels for different parts of an article or document. - Experience replay: A technique used in training programs to help them remember things they have learned before.

Introducing MEDITRON: An Open-Source Suite of Large Language Models for the Medical Domain

In this work, researchers introduce MEDITRON, a suite of open-source large language models (LLMs) specifically designed for the medical domain. These LLMs have 7B and 70B parameters and are built upon Llama-2 using Nvidia's Megatron-LM distributed trainer. To train these models, a comprehensively curated medical corpus was used, which includes selected PubMed articles, abstracts, and internationally-recognized medical guidelines.

The GUIDELINES Corpus

The GUIDELINES corpus included in the training data covers a wide range of contexts in terms of geographic scope (from global to institutional), resource settings (high-, low-, and volatile-resource settings), and target audiences (clinicians or patients). It also incorporates various peer review processes ranging from UN bodies to publicly crowdsourced knowledge bases.

Preprocessing Content from PubMed Sources

To create the pre-training mix for MEDITRON, PubMed papers and abstracts were chosen due to their vast amount of biomedical textual data. The researchers collected 4.47M full-text papers from the PubMed Central Open Access Subset as well as 444,521 open-access full-text PubMed papers that were not found in the archive. Additionally, 16,209,047 PubMed and PubMed Central abstracts were collected with a knowledge cutoff date of August 2023. To preprocess the content from these sources metadata information references acknowledgments tables figures URLs and non-English content were removed while inline citations section headers figures tables mathematical formulas were identified using special tokens. In-text references were formatted with a similar methodology to the Galactica model to promote accurate citations by replacing them with [BIB_REF] tokens containing truncated titles and authors' last names while figure and table references were wrapped with [FIG_REF] tokens containing figure numbers and truncated captions. Section headers were indicated with '#'and '##' tokens for main sections and subsections respectively.

Experience Replay Training Process

Experience replay was employed in the training process to overcome catastrophic forgetting which helps prevent the model from forgetting previously learned information by including data from old tasks when training on new tasks