# CP2: Copy-Paste Contrastive Pretraining for Semantic Segmentation

**Authors:** Feng Wang, Huiyu Wang, Chen Wei, Alan Yuille, Wei Shen  
**arXiv:** [2203.11709](https://arxiv.org/abs/2203.11709)  
**Date:** 2022-03-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.11709v2/

## Key points

- Recent advances in self-supervised contrastive learning for image-level representation learning
- Limitation of overlooking pixel-level detailed information in dense prediction tasks
- Introduction of CP2 (Copy-Paste Contrastive Pretraining) method
- CP2 facilitates image and pixel-level representation learning
- Copying foreground from an image and pasting onto various backgrounds
- Pretraining semantic segmentation model with two objectives: distinguishing foreground from background pixels and identifying composed images with the same foreground
- Strong performance of CP2 on downstream semantic segmentation tasks
- Impressive results achieved with fine-tuned CP2 pretrained models on PASCAL VOC 2012 dataset (ResNet-50: 78.6% mIoU, ViT-S: 79.5% mIoU)
- Addressing limitations of existing self-supervised contrastive learning methods in dense prediction tasks

## Summary

Recent advances in self-supervised contrastive learning have shown promising results for image-level representation learning in classification tasks. However, these methods often overlook pixel-level detailed information, leading to suboptimal performance in dense prediction tasks such as semantic segmentation. To address this limitation, the authors propose a novel pixel-wise contrastive learning method called CP2 (Copy-Paste Contrastive Pretraining). CP2 facilitates both image and pixel-level representation learning to improve transfer performance for downstream dense prediction tasks. The approach involves copying a random crop from an image (the foreground) and pasting it onto various background images. A semantic segmentation model is then pretrained with two objectives: 1) distinguishing the foreground pixels from the background pixels; and 2) identifying composed images that share the same foreground. Experimental results demonstrate the strong performance of CP2 on downstream semantic segmentation tasks. By fine-tuning CP2 pretrained models on PASCAL VOC 2012 dataset, impressive results were achieved with a ResNet-50 model achieving 78.6% mean Intersection over Union (mIoU), while a ViT-S model achieved 79.5% mIoU. Overall, CP2 addresses the limitations of existing self-supervised contrastive learning methods by incorporating both image and pixel level information to improve transfer performance in dense prediction tasks such as semantic segmentation.