# Customizing General-Purpose Foundation Models for Medical Report Generation

**Authors:** Bang Yang, Asif Raza, Yuexian Zou, Tong Zhang  
**arXiv:** [2306.05642](https://arxiv.org/abs/2306.05642)  
**Date:** 2023-06-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.05642v1/

## Key points

- Medical report generation (MRG) involves automatically generating accurate and coherent captions for medical images.
- Scarcity of labeled medical image-report pairs poses challenges in developing deep and large-scale neural networks for MRG.
- The authors propose customizing off-the-shelf general-purpose large-scale pre-trained models, known as foundation models (FMs), for MRG.
- Their encoder-decoder based MRG model utilizes a lightweight query Transformer to connect two FMs: EVA-ViT-g (vision Transformer) and ChatGLM-6B (bilingual language model).
- Unfreezing EVA-ViT-g to learn medical image representations and parameter efficient training of ChatGLM 6B are crucial factors for optimal results.
- The authors achieved impressive rankings in the ImageCLEFmedical Caption 2023 competition based on BERTScore and ROUGE 1 metrics.
- Previous research on MRG has focused on cross modal alignment, reinforcement learning, architecture design, explicit loss constraints, retrieval, and knowledge augmented approaches.
- Foundation models have become a research hotspot in computer vision and natural language processing.
- Prompt engineering and parameter efficient transfer learning are popular techniques in leveraging foundation models.
- This work presents a novel approach to MRG by customizing off-the-shelf foundation models, with experimental results demonstrating its effectiveness.

## Summary

Medical report generation (MRG) is a task that involves automatically generating accurate and coherent captions for medical images. However, the scarcity of labeled medical image-report pairs poses challenges in developing deep and large-scale neural networks capable of harnessing the power of artificial general intelligence. In this work, the authors propose customizing off-the-shelf general-purpose large-scale pre-trained models, known as foundation models (FMs), in computer vision and natural language processing for MRG. The authors introduce their encoder-decoder based MRG model which utilizes a lightweight query Transformer to connect two FMs: EVA-ViT-g, a giant vision Transformer; and ChatGLM-6B, a bilingual large language model trained to align with human intentions. They conduct ablative experiments on the trainable components of the model to identify crucial factors for effective transfer learning. The findings demonstrate that unfreezing EVA-ViT-g to learn medical image representations and parameter efficient training of ChatGLM 6B to capture writing styles of medical reports are essential for optimal results. The authors participated in the ImageCLEFmedical Caption 2023 Caption Prediction Task competition and achieved impressive rankings based on BERTScore and ROUGE 1 metrics. This highlights the effectiveness of their approach in generating high quality medical captions. In related works, previous research on medical report generation has mainly focused on improving cross modal alignment between images and reports through reinforcement learning, architecture design or explicit loss constraints. Other approaches explore retrieval and knowledge augmented report generation but these methods are limited by the small number of labeled pairs available for training. The adaptation of foundation models has become a research hotspot in computer vision and natural language processing. Techniques such as prompt engineering aim to influence the behaviors of language models by providing them with task related priors or examples while another popular technique is parameter efficient transfer learning. Overall this work presents a novel approach to medical report generation by leveraging off the shelf foundation models and customizing them for the task. The experimental results demonstrate the effectiveness of their model in generating accurate and coherent medical captions showcasing its potential for addressing challenges in this field.