Results of the summarizing process for the arXiv paper: 2303.17602v1

Human-centric visual tasks have become increasingly important due to their wide-ranging applications. To address the need for a general human representation from massive unlabeled human images that can benefit downstream human-centric tasks, this paper proposes a Semantic cOntrollable seLf-supervIseD lEaRning framework (SOLIDER). Unlike existing self-supervised learning methods, SOLIDER utilizes prior knowledge from human images to build pseudo semantic labels and import more semantic information into the learned representation. The framework also recognizes that different downstream tasks require varying ratios of semantic and appearance information. For instance, while human parsing requires more semantic information, person re-identification needs more appearance information for identification purposes. To solve this problem, SOLIDER introduces a conditional network with a semantic controller. After training the model, users can send values to the controller to produce representations with different ratios of semantic information that fit various downstream task requirements. In addition to outperforming state-of-the-art methods on six downstream human-centric visual tasks, SOLIDER builds new baselines for these tasks. The proposed framework extends the representation to different downstream human-centric visual tasks by training it with more semantic information than appearance information. Prior knowledge from human images is utilized in SOLIDER to discover semantic information and produce pseudo-semantic labels for every token. The paper also discusses how SOLIDER utilizes a pretext task based on predicting token order as an auxiliary task during training to improve performance further. Finally, the authors demonstrate how SOLIDER can be used in several applications such as image captioning, action recognition, object detection and segmentation, human parsing and pose estimation. The code is available at https://github.com/tinyvision/SOLIDER.

• Human-centric visual tasks are important and have many applications
• The paper proposes a framework called SOLIDER for creating a general human representation from unlabeled images
• SOLIDER uses prior knowledge from human images to build pseudo semantic labels and import more semantic information into the learned representation
• Different downstream tasks require varying ratios of semantic and appearance information, which SOLIDER addresses through a conditional network with a semantic controller
• SOLIDER outperforms state-of-the-art methods on six downstream human-centric visual tasks and builds new baselines for these tasks
• The framework can be used in several applications such as image captioning, action recognition, object detection and segmentation, human parsing and pose estimation
• A pretext task based on predicting token order is utilized during training to improve performance further

Introducing SOLIDER: A Semantic cOntrollable seLf-supervIseD lEaRning Framework for Human-Centric Visual Tasks

Human-centric visual tasks have become increasingly important due to their wide range of applications. To address the need for a general human representation from massive unlabeled human images that can benefit downstream human-centric tasks, this paper proposes a novel Semantic cOntrollable seLf-supervIseD lEaRning framework (SOLIDER). This framework is designed to extend the representation to different downstream human-centric visual tasks by training it with more semantic information than appearance information.

What is SOLIDER?

SOLIDER is a self-supervised learning method that utilizes prior knowledge from human images to build pseudo semantic labels and import more semantic information into the learned representation. Unlike existing self-supervised learning methods, SOLIDER recognizes that different downstream tasks require varying ratios of semantic and appearance information. For instance, while human parsing requires more semantic information, person re-identification needs more appearance information for identification purposes. To solve this problem, SOLIDER introduces a conditional network with a semantic controller. After training the model, users can send values to the controller to produce representations with different ratios of semantic information that fit various downstream task requirements.

How Does It Work?

The proposed framework consists of two components: an encoder network and a decoder network which are connected by an attention mechanism based on token order prediction as an auxiliary task during training. The encoder takes in input images and extracts tokens which are then used by the decoder network for predicting token order as well as generating pseudo labels using prior knowledge from human images such as body parts or facial features. The decoder also has access to additional contextual cues such as object cooccurrence or scene context which helps improve performance further. Finally, users can control how much semantic versus appearance information is included in each token by sending values to the controller before feeding them into downstream tasks such as image captioning, action recognition, object detection and segmentation, human parsing and pose estimation among others.

Results & Applications

In addition to outperforming state-of-the art methods on six downstream human centric visual tasks including image captioning, action recognition etc., SOLIDER builds new baselines for these tasks too! Furthermore ,the code is available at https://github/tinyvision/SOLIDER .

Conclusion

This paper presents SOLIDER – a novel Semantic cOntrollable seLf superviseD lEarning framework designed specifically for addressing problems related to building general representations from massive unlabeled datasets that can be used in various downstream applications involving humans such as image captioning or person reidentification etc.. By utilizing prior knowledge from human images along with introducing a conditional network with a semantic controller ,SOLIDE allows users greater flexibility when selecting appropriate representations depending upon their specific application needs .