# MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining

**Authors:** Di Wang, Jing Zhang, Minqiang Xu, Lin Liu, Dongsheng Wang, Erzhong Gao, Chengxi Han, Haonan Guo, Bo Du, Dacheng Tao, Liangpei Zhang  
**arXiv:** [2403.13430](https://arxiv.org/abs/2403.13430)  
**Date:** 2024-03-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.13430v1/

## Key points

- Study titled "MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining" by Di Wang, Jing Zhang, Minqiang Xu, Lin Liu, Dongsheng Wang, Erzhong Gao, Chengxi Han, Haonan Guo, Bo Du, Dacheng Tao, and Liangpei Zhang
- Focus on enhancing image interpretation tasks in Remote Sensing (RS) using foundation models
- Introduction of Multi-Task Pretraining (MTP) paradigm to address task discrepancy during model transfer to downstream tasks
- Utilization of shared encoder and task-specific decoder architecture for multi-task supervised pretraining on the SAMRS dataset
- Support for convolutional neural networks and vision transformer foundation models with over 300 million parameters
- Fine-tuning of pretrained models on various RS downstream tasks leading to outperformance of existing models and competitive performance compared to larger state-of-the-art models

## Summary

, , , , 

In their study titled "MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining," authors Di Wang, Jing Zhang, Minqiang Xu, Lin Liu, Dongsheng Wang, Erzhong Gao, Chengxi Han, Haonan Guo, Bo Du, Dacheng Tao, and Liangpei Zhang delve into the realm of Remote Sensing (RS) and the significant impact foundation models have had on enhancing image interpretation tasks. The focus of their research lies in pretraining methods for these models using supervised and self-supervised learning techniques to effectively initialize model weights. To address this issue of task discrepancy during model transfer to downstream tasks such as image classification or object discrimination tasks, the authors propose the Multi-Task Pretraining (MTP) paradigm for RS foundation models. This approach utilizes a shared encoder and task-specific decoder architecture to conduct multi-task supervised pretraining on the SAMRS dataset. Tasks included in this phase encompass semantic segmentation, instance segmentation, and rotated object detection. Notably, MTP supports both convolutional neural networks and vision transformer foundation models with over 300 million parameters. Following the multi-task supervised pretraining phase on SAMRS dataset, the pretrained models are fine-tuned on various RS downstream tasks including scene classification, horizontal and rotated object detection, semantic segmentation, and change detection. Through extensive experiments conducted across 14 datasets,<fd>the authors demonstrate that their models outperform existing ones of similar size.</fd> Furthermore,<fd>they also showcase competitive performance compared to larger state-of-the-art models,</fd> validating the effectiveness of MTP in optimizing model performance for complex image interpretation tasks within the field of Remote Sensing. Overall, this research highlights the significance of Multi-Task Pretraining in advancing Remote Sensing Foundation Models by addressing task discrepancy issues encountered during model transfer to downstream tasks. The findings underscore the importance of innovative pretraining paradigms in optimizing model performance for complex image interpretation tasks within the field of Remote Sensing.