# M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization

**Authors:** Che Liu, Sibo Cheng, Chen Chen, Mengyun Qiao, Weitong Zhang, Anand Shah, Wenjia Bai, Rossella Arcucci  
**arXiv:** [2307.08347](https://arxiv.org/abs/2307.08347)  
**Date:** 2023-07-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.08347v2/

## Key points

- Integration of features from medical imaging and clinical text in vision-language models is challenging due to training complexity and latent space navigation.
- M-FLAG (Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization) addresses these challenges by leveraging a frozen language model and introducing an orthogonality loss for latent space geometry harmonization.
- M-FLAG demonstrates significant improvements over existing methods in medical image classification, segmentation, and object detection tasks while reducing parameter count by 78%.
- Excels in segmentation task even with minimal training data, outperforming ImageNet pre-trained models fine-tuned with full datasets.
- Offers simplicity, efficiency, low computational costs, and stable training processes through its architecture utilizing a frozen language model and orthogonality loss function.
- Represents a major advancement in medical vision-language modeling by providing superior performance metrics, reduced parameter complexity, and robust adaptability across diverse tasks.

## Summary

, , , , 

In the field of medical vision-language models, the integration of features from medical imaging and clinical text has proven to be a challenging task. This is due to the complexity of training these models and navigating the latent representation space. To address these challenges, a novel approach called Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization (M-FLAG) has been proposed. M-FLAG leverages a frozen language model for enhanced training stability and efficiency, while introducing an innovative orthogonality loss to harmonize the geometry of the latent space. The potential of M-FLAG was demonstrated through extensive experiments on three key downstream tasks: medical image classification, segmentation, and object detection. The results showcased significant improvements over existing medical vision-language pre-training methods, with M-FLAG achieving remarkable performance while reducing the number of parameters by 78%. Notably, M-FLAG excelled in the segmentation task even when trained on only 1% of the RSNA dataset, surpassing ImageNet pre-trained models fine-tuned with 100% of the data. Furthermore, M-FLAG's architecture offers simplicity and efficiency which contributes to low computational costs and stable training processes. Its success lies in its utilization of a frozen language model alongside a latent space orthogonality loss function. These elements work together to enhance performance across various tasks and datasets, showcasing M-FLAG's robustness when transferred to unseen test sets. Overall,<Organization> M-FLAG represents a significant advancement in medical vision-language modeling by offering superior performance metrics, reduced parameter complexity, and robust adaptability across diverse downstream tasks. Its success underscores the importance of freezing language models and implementing effective regularization techniques in optimizing model performance within complex medical imaging contexts.