# BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization

**Authors:** Weichao Zhao, Hezhen Hu, Wengang Zhou, Jiaxin Shi, Houqiang Li  
**arXiv:** [2302.05075](https://arxiv.org/abs/2302.05075)  
**Date:** 2023-02-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.05075v3/

## Key points

- Authors propose a method called BEST to improve sign language recognition using BERT pre-training
- BEST leverages the success of BERT pre-training and model domain-specific statistics
- Hand and body movements are organized as pose triplet units and fed into the Transformer backbone in a frame-wise manner
- Pre-training involves reconstructing masked triplet units from corrupted input sequences to learn hierarchical correlation context cues
- Coupling tokenization is introduced to bridge the semantic gap between low-level pose units and high-level semantics required for SLR tasks
- After pre-training, fine-tuning is performed on downstream SLR tasks with a newly added task-specific layer
- Extensive experiments validate the proposed method, achieving new state-of-the-art performance on four benchmarks
- RGB-based methods and pose-based methods have been studied extensively in sign language recognition
- BEST addresses limitations of existing approaches by leveraging BERT pre-training and incorporating domain-specific statistics through pose triplet units
- The method achieves improved performance compared to previous state-of-the-art methods

## Summary

In this work, the authors propose a method called BEST (BERT Pre-Training for Sign Language Recognition with Coupling Tokenization) to improve sign language recognition (SLR) using BERT pre-training. They leverage the success of BERT pre-training and model domain-specific statistics to enhance the SLR model. Since hand and body movements are dominant in sign language expression, they organize them as pose triplet units and feed them into the Transformer backbone in a frame-wise manner. To perform pre-training, the authors reconstruct the masked triplet unit from the corrupted input sequence. This helps learn hierarchical correlation context cues among internal and external triplet units. However, unlike BERT's highly semantic word tokens, pose units are low-level signals originally located in continuous space. This prevents the direct adoption of BERT's cross-entropy objective. To bridge this semantic gap, they introduce coupling tokenization of the triplet unit. This adaptively extracts discrete pseudo labels from the pose triplet unit, representing semantic gesture/body states. After pre-training, the authors fine-tune the pre-trained encoder on downstream SLR tasks along with a newly added task-specific layer. Extensive experiments are conducted to validate their proposed method, achieving new state-of-the-art performance on four benchmarks with a notable gain. In related work, sign language recognition has been studied extensively in recent years using RGB-based methods and pose-based methods. RGB based methods focus on visual information captured by cameras while pose based methods analyze body joint positions obtained from depth sensors or 2D/3D poses estimated from RGB images. The proposed BEST method addresses some limitations of existing approaches by leveraging BERT pre training and incorporating domain specific statistics through pose triplet units. By bridging the semantic gap between low level pose units and high level semantics required for SLR tasks their method achieves improved performance compared to previous state of the art methods. Overall this work contributes to advancing sign language recognition by effectively utilizing pre training techniques and modeling domain specific statistics leading to more accurate and robust SLR models.