# Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks

**Authors:** Weihua Chen, Xianzhe Xu, Jian Jia, Hao luo, Yaohua Wang, Fan Wang, Rong Jin, Xiuyu Sun  
**arXiv:** [2303.17602](https://arxiv.org/abs/2303.17602)  
**Date:** 2023-03-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.17602v1/

## Key points

- Human-centric visual tasks are important and have many applications
- The paper proposes a framework called SOLIDER for creating a general human representation from unlabeled images
- SOLIDER uses prior knowledge from human images to build pseudo semantic labels and import more semantic information into the learned representation
- Different downstream tasks require varying ratios of semantic and appearance information, which SOLIDER addresses through a conditional network with a semantic controller
- SOLIDER outperforms state-of-the-art methods on six downstream human-centric visual tasks and builds new baselines for these tasks
- The framework can be used in several applications such as image captioning, action recognition, object detection and segmentation, human parsing and pose estimation
- A pretext task based on predicting token order is utilized during training to improve performance further

## Summary

Human-centric visual tasks have become increasingly important due to their wide-ranging applications. To address the need for a general human representation from massive unlabeled human images that can benefit downstream human-centric tasks, this paper proposes a Semantic cOntrollable seLf-supervIseD lEaRning framework (SOLIDER). Unlike existing self-supervised learning methods, SOLIDER utilizes prior knowledge from human images to build pseudo semantic labels and import more semantic information into the learned representation. The framework also recognizes that different downstream tasks require varying ratios of semantic and appearance information. For instance, while human parsing requires more semantic information, person re-identification needs more appearance information for identification purposes. To solve this problem, SOLIDER introduces a conditional network with a semantic controller. After training the model, users can send values to the controller to produce representations with different ratios of semantic information that fit various downstream task requirements. In addition to outperforming state-of-the-art methods on six downstream human-centric visual tasks, SOLIDER builds new baselines for these tasks. The proposed framework extends the representation to different downstream human-centric visual tasks by training it with more semantic information than appearance information. Prior knowledge from human images is utilized in SOLIDER to discover semantic information and produce pseudo-semantic labels for every token. The paper also discusses how SOLIDER utilizes a pretext task based on predicting token order as an auxiliary task during training to improve performance further. Finally, the authors demonstrate how SOLIDER can be used in several applications such as image captioning, action recognition, object detection and segmentation, human parsing and pose estimation. The code is available at https://github.com/tinyvision/SOLIDER.