# Enlarging Instance-specific and Class-specific Information for Open-set Action Recognition

**Authors:** Jun Cen, Shiwei Zhang, Xiang Wang, Yixuan Pei, Zhiwu Qing, Yingya Zhang, Qifeng Chen  
**arXiv:** [2303.15467](https://arxiv.org/abs/2303.15467)  
**Date:** 2023-03-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.15467v1/

## Key points

- Open-set action recognition (OSAR) involves rejecting unknown human action cases that fall outside the distribution of the training set.
- Existing methods for OSAR focus on learning better uncertainty scores, but they often overlook the importance of feature representations.
- The authors propose a novel Prototypical Similarity Learning (PSL) framework to enlarge instance-specific (IS) and class-specific (CS) information in feature representations for better OSAR performance.
- CS information is used for inter-class recognition, while IS information is unique to each sample within a class. Both types of information are crucial for OSAR performance.
- To enlarge IS information, PSL encourages instances to have less than 1 similarity with their corresponding prototypes, retaining more IS information in learned feature representations.
- Video shuffling is introduced into PSL to alleviate misclassification issues caused by OoD videos that share similar appearances with InD videos.
- Shuffled videos are encouraged to have less than 1 similarity with original samples, allowing networks to extract distinct temporal information among them and enlarging CS information.
- Experiments demonstrate that PSL significantly boosts both open-set and closed-set performance on multiple benchmarks, achieving state-of-the-art results.
- The proposed framework provides a novel perspective on analyzing OSAR tasks based on the information bottleneck theory and highlights the importance of retaining both IS and CS information for optimal performance.

## Summary

The problem of open-set action recognition (OSAR) involves rejecting unknown human action cases that fall outside the distribution of the training set. Existing methods for OSAR focus on learning better uncertainty scores, but they often overlook the importance of feature representations. This paper proposes a novel Prototypical Similarity Learning (PSL) framework to enlarge instance-specific (IS) and class-specific (CS) information in feature representations for better OSAR performance. The authors begin by analyzing the behavior of feature representations in the open-set problem using the information bottleneck theory. They divide the information contained in features into IS and CS categories. CS information is used for inter-class recognition, while IS information is unique to each sample within a class. Both types of information are crucial for OSAR performance. To enlarge IS information, PSL encourages instances to have less than 1 similarity with their corresponding prototypes, retaining more IS information in learned feature representations. The authors also introduce video shuffling into PSL to alleviate misclassification issues caused by OoD videos that share similar appearances with InD videos. Shuffled videos are encouraged to have less than 1 similarity with original samples, allowing networks to extract distinct temporal information among them and enlarging CS information. Experiments demonstrate that PSL significantly boosts both open-set and closed-set performance on multiple benchmarks, achieving state-of-the-art results. The proposed framework provides a novel perspective on analyzing OSAR tasks based on the information bottleneck theory and highlights the importance of retaining both IS and CS information for optimal performance.