# Dense Contrastive Learning for Self-Supervised Visual Pre-Training

**Authors:** Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, Lei Li  
**arXiv:** [2011.09157](https://arxiv.org/abs/2011.09157)  
**Date:** 2020-11-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2011.09157v2/

## Key points

- The paper addresses limitations of existing self-supervised learning methods for image classification tasks
- Existing pre-trained models often fail in dense prediction tasks due to disparity between image-level and pixel-level predictions
- Proposed approach called dense contrastive learning focuses on pixel-level features and their correspondence
- Introduces pairwise contrastive (dis)similarity loss at the pixel level between two views of input images
- Achieves self-supervised learning and captures local feature correspondences effectively
- Incurs negligible computational overhead compared to baseline method MoCo-v2
- Outperforms MoCo-v2 in downstream dense prediction tasks such as object detection, semantic segmentation, and instance segmentation
- Significant improvements over MoCo-v2 baseline: 2.0% AP improvement on PASCAL VOC object detection, 1.1% AP improvement on COCO object detection, 0.9% AP improvement on COCO instance segmentation, 3.0% mIoU improvement on PASCAL VOC semantic segmentation, and 1.8% mIoU improvement on Cityscapes semantic segmentation
- Provides an effective solution for self-supervised visual pre-training by considering pixel-level features and their correspondence directly

## Summary

The paper "Dense Contrastive Learning for Self-Supervised Visual Pre-Training" addresses the limitations of existing self-supervised learning methods that are primarily designed for image classification tasks. These pre-trained models often fail to perform optimally in dense prediction tasks due to the disparity between image-level and pixel-level predictions. To overcome this gap, the authors propose a novel approach called dense contrastive learning, which focuses on pixel-level features and their correspondence. The method introduces a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images. By optimizing this loss function, the proposed method achieves self-supervised learning and effectively captures local feature correspondences. Compared to the baseline method MoCo-v2, the proposed approach incurs negligible computational overhead (less than 1% slower) while consistently outperforming it in downstream dense prediction tasks such as object detection, semantic segmentation, and instance segmentation. The experimental results demonstrate significant improvements over MoCo-v2 baseline, with 2.0% average precision (AP) improvement on PASCAL VOC object detection, 1.1% AP improvement on COCO object detection, 0.9% AP improvement on COCO instance segmentation, 3.0% mean intersection over union (mIoU) improvement on PASCAL VOC semantic segmentation, and 1.8% mIoU improvement on Cityscapes semantic segmentation. Overall, the proposed dense contrastive learning method provides an effective solution for self-supervised visual pre-training by directly considering pixel-level features and their correspondence. The code for implementing this method is available at https://git.io/AdelaiDet . Authors: Xinlong Wang , Rufeng Zhang , Chunhua Shen , Tao Kong , Lei Li . Title: Dense Contrastive Learning for Self - Supervised Visual Pre - Training .