# R2GenGPT: Radiology Report Generation with Frozen LLMs

**Authors:** Zhanyu Wang, Lingqiao Liu, Lei Wang, Luping Zhou  
**arXiv:** [2309.09812](https://arxiv.org/abs/2309.09812)  
**Date:** 2023-09-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.09812v1/

## Key points

- Radiological imaging data is growing rapidly, leading to an overwhelming workload for radiologists.
- Surge in volume and complexity of cases puts pressure on radiologists to interpret more studies within tight timeframes.
- Automated radiographic report generation (R2Gen) systems can alleviate the burden on radiologists, reduce errors, and expedite clinical workflows.
- Different approaches to R2Gen include structured and template-based methods.
- This paper focuses on unstructured multi-sentence report generation.
- Most methodologies in medical report generation are inspired by image/video captioning and adopt the encoder-decoder paradigm with improvements tailored to R2Gen.
- Two major challenges in R2Gen are long text generation and bias present in visual and textual data used for training models.
- Solutions for long text generation include hierarchically structured LSTM models and memory-driven Transformers.
- Bias towards normal samples in training data can affect model performance when generating reports for abnormal cases.
- R2GenGPT is a novel solution that aligns visual features with the word embedding space of large language models (LLMs) using an efficient visual alignment module.
- R2GenGPT allows LLMs to seamlessly integrate and process image information, leading to improved R2Gen performance.

## Summary

The field of radiological imaging data is growing rapidly, leading to an overwhelming workload for radiologists. This surge in volume and complexity of cases puts pressure on radiologists to interpret more studies within tight timeframes, resulting in extended working hours and increased risk of diagnostic errors. To address this issue, there is a growing demand for automated radiographic report generation (R2Gen) systems that can alleviate the burden on radiologists, reduce errors, and expedite clinical workflows. Automated R2Gen is a complex AI task that aims to generate coherent paragraphs capturing observations and findings from radiology images. There are different approaches to R2Gen, including structured and template-based methods. This paper focuses on unstructured multi-sentence report generation. The field of medical report generation has been gaining attention due to its clinical relevance. Most methodologies in this field are inspired by image/video captioning and adopt the encoder-decoder paradigm with improvements tailored to the unique characteristics of R2Gen. Recent works in R2Gen primarily aim to tackle two major challenges. The first challenge is long text generation. Unlike image captioning tasks that generate single-sentence descriptions, medical report generation requires detailed and coherent paragraph-long descriptions. To address this challenge, various solutions have been proposed such as hierarchically structured LSTM models that produce topic vectors using a sentence LSTM and create descriptions for each generated topic with a word LSTM. Another approach involves memory-driven Transformers that can record key information during the generation process, enhancing the model's ability to produce long texts. The second challenge lies in the bias present in visual and textual data used for training R2Gen models. Due to an over-representation of normal samples in the training data, models tend to be biased towards these samples. This bias can affect the model's performance when generating reports for abnormal cases. To bridge the gap between large language models (LLMs) and R2Gen tasks effectively, this paper proposes R2GenGPT, a novel solution that aligns visual features with the word embedding space of LLMs using an efficient visual alignment module. This approach allows the previously static LLM to seamlessly integrate and process image information, leading to improved R2Gen performance. R2GenGPT offers several benefits.