# Learnt Contrastive Concept Embeddings for Sign Recognition

**Authors:** Ryan Wong, Necati Cihan Camgoz, Richard Bowden  
**arXiv:** [2308.09515](https://arxiv.org/abs/2308.09515)  
**Date:** 2023-08-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.09515v1/

## Key points

- Sign recognition has seen various approaches, from hand-crafted features to data-driven methods.
- Bridging the gap between sign language and spoken language is a common challenge in sign recognition.
- Word embeddings have been useful in encoding the meaning of words in spoken languages, but there is a need for sign embeddings that capture visual and linguistic semantics of sign languages.
- The authors propose a learning framework to derive Learnt Contrastive Concept (LCC) embeddings for sign language.
- The focus is on creating sign embeddings that bridge the gap between sign language and spoken language.
- Weakly supervised contrastive learning is used to train a vocabulary of embeddings based on linguistic labels for sign videos.
- A conceptual similarity loss leverages word embeddings from NLP methods to create sign embeddings with better correspondence between sign language and spoken language.
- These learned representations encode the meaning of signs and enable automatic localization of signs in time.
- Experiments on two large-scale datasets (WLASL and BOBSL) show that the proposed approach achieves state-of-the-art performance in keypoint-based sign recognition tasks.
- Prior research has explored different strategies, such as using hand and mouthing shapes as features or specialized classification models, but these often require manual annotation or specialized models.
- Large-scale datasets like RWTH-PHOENIX-Weather have played a crucial role in advancing deep learning-based approaches for sign recognition.

## Summary

The field of sign recognition has seen various approaches over the years, ranging from hand-crafted features to data-driven methods. One common challenge in sign recognition is bridging the gap between sign language and spoken language. While word embeddings have proven useful in encoding the meaning of words in natural language processing (NLP) of spoken languages, there is a need for sign embeddings that can capture the visual and linguistic semantics of sign languages. In this study, the authors propose a learning framework to derive Learnt Contrastive Concept (LCC) embeddings for sign language. Unlike many existing approaches, their focus is on explicitly creating sign embeddings that can bridge the gap between sign language and spoken language. The proposed approach utilizes weakly supervised contrastive learning to train a vocabulary of embeddings based on linguistic labels for sign videos. Additionally, the authors introduce a conceptual similarity loss that leverages word embeddings from NLP methods. This allows them to create sign embeddings with better correspondence between sign language and spoken language. These learned representations not only encode the meaning of signs but also enable automatic localization of signs in time. The effectiveness of the proposed approach is demonstrated through experiments on two large-scale datasets: WLASL and BOBSL. The results show that their approach achieves state-of-the-art performance in keypoint-based sign recognition tasks. Prior research has explored different strategies for solving these tasks, including breaking down the problem into subproblems by using hand and mouthing shapes as features; however, these approaches often require manual annotation at frame level or specialized classification models. Large-scale datasets have played a crucial role in advancing deep learning-based approaches for sign recognition; for instance, datasets like RWTH-PHOENIX-Weather-2014 and RWTH-PHOENIX-Weather 2014T have been used to predict signs in videos with models trained with Connectionist Temporal Classification (CTC) loss being successful in tackling this task. Overall, this study contributes to the development of sign recognition by proposing a novel approach that explicitly creates sign embeddings and leverages conceptual similarity loss. The results demonstrate the effectiveness of their approach in achieving state-of-the art performance in keypoint basedsign recognition tasks.