AI summary · arXiv: 2302.05075v3 · cs.CV
BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization
The essentials
- Authors propose a method called BEST to improve sign language recognition using BERT pre-training
- BEST leverages the success of BERT pre-training and model domain-specific statistics
- Hand and body movements are organized as pose triplet units and fed into the Transformer backbone in a frame-wise manner
- Pre-training involves reconstructing masked triplet units from corrupted input sequences to learn hierarchical correlation context cues
- Coupling tokenization is introduced to bridge the semantic gap between low-level pose units and high-level semantics required for SLR tasks
- After pre-training, fine-tuning is performed on downstream SLR tasks with a newly added task-specific layer
- Extensive experiments validate the proposed method, achieving new state-of-the-art performance on four benchmarks
- RGB-based methods and pose-based methods have been studied extensively in sign language recognition
- BEST addresses limitations of existing approaches by leveraging BERT pre-training and incorporating domain-specific statistics through pose triplet units
- The method achieves improved performance compared to previous state-of-the-art methods
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language RecognitionHe-Zhen Hu, Wei-Chao Zhao, Wen-Gang Zhou et al. · 2021 · IEEE International Conference on Computer Vision · cited 154 times · Not summarized yet
Abstract
Hand gesture serves as a critical role in sign language. Current deep-learning-based sign language recognition (SLR) methods may suffer insufficient interpretability and overfitting due to limited sign data sources. In this paper, we introduce the first self-supervised pre-trainable SignBERT with incorporated hand prior for SLR. Sign-BERT views the hand pose as a visual token, which is derived from an off-the-shelf pose extractor. The visual tokens are then embedded with gesture state, temporal and hand chirality information. To take full advantage of available sign data sources, SignBERT first performs self-supervised pre-training by masking and reconstructing visual tokens. Jointly with se…
-
SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language UnderstandingHe-Zhen Hu, Wei-Chao Zhao, Wen-Gang Zhou et al. · 2023 · IEEE Transactions on Pattern Analysis and Machine Intelligence · cited 195 times · Not summarized yet
Abstract
Hand gesture serves as a crucial role during the expression of sign language. Current deep learning based methods for sign language understanding (SLU) are prone to over-fitting due to insufficient sign data resource and suffer limited interpretability. In this paper, we propose the first self-supervised pre-trainable SignBERT+ framework with model-aware hand prior incorporated. In our framework, the hand pose is regarded as a visual token, which is derived from an off-the-shelf detector. Each visual token is embedded with gesture state and spatial-temporal position encoding. To take full advantage of current sign data resource, we first perform self-supervised learning to model its statisti…
-
Pose-based Sign Language Recognition using GCN and BERTA. Tunga, Sai Vidyaranya Nuthalapati, J. Wachs · 2020 · 2021 IEEE Winter Conference on Applications of Computer Vision Workshops (WACVW) · cited 121 times · Not summarized yet
Abstract
Sign language recognition (SLR) plays a crucial role in bridging the communication gap between the hearing and vocally impaired community and the rest of the society. Word-level sign language recognition (WSLR) is the first important step towards understanding and interpreting sign language. However, recognizing signs from videos is a challenging task as the meaning of a word depends on a combination of subtle body motions, hand configurations and other movements. Recent pose-based architectures for WSLR either model both the spatial and temporal dependencies among the poses in different frames simultaneously or only model the temporal information without fully utilizing the spatial informat…
-
SignBERT: A BERT-based Deep Learning Framework for Continuous Sign Language RecognitionZhenxing Zhou, Vincent W. L. Tam, Edmund Y. Lam · 2021 · IEEE Access · cited 58 times · Not summarized yet
Abstract
Continuous sign language recognition (CSLR) is a very challenging task in intelligent systems, since it requires to produce real-time responses while performing computationally intensive video analytics and language modeling. Previous studies mainly focus on adopting hidden Markov models or recurrent neural networks with a limited capability to model specific sign languages, and the accuracy can drop significantly when recognizing the signs performed by different signers with non-standard gestures or non-uniform speeds. In this work, we develop a deep learning framework named SignBERT, integrating the bidirectional encoder representations from transformers (BERT) with the residual neural net…
-
HBT-S2S: A Hybrid BiLSTM–Transformer Architecture for Efficient Landmark-Based Sign-to-Speech RecognitionMd Rakibul Islam Midul, Tahmid Ul Haque Choudhury, Misbahul Karim et al. · 2026 · International Workshop on Artificial Intelligence and Cognition · Not summarized yet
Abstract
Sign Language Recognition (SLR) is critical in ensuring effective communication between hearing-impaired communities and hearing people. Current methods largely rely on time-consuming RGB-based systems, making their practical implementation and deployment limited in resource-constrained environments. In this study, we propose HBT-S2S, a new hybrid bi-directional LSTM (BiLSTM)-Transformer model for recognizing signs using only low-cost 2D pose, hand, and facial landmarks. What makes this method unique is the ability of capturing both local and global movements in a lightweight structure without the use of costly visual features. The stacked bidirectional LSTM captures small-scale dynamics, wh…
-
Scaling up Multimodal Pre-Training for Sign Language UnderstandingWen-Gang Zhou, Wei-Chao Zhao, He-Zhen Hu et al. · 2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence · cited 36 times · Not summarized yet
Abstract
Sign language pre-training (SLP) has significantly improved the performance of diverse sign language understanding (SLU) tasks. However, many existing methods employ pre-training techniques that are tailored to a specific task with small data scale, resulting in limited model generalization. Some others focus solely on exploring visual cues, neglecting semantically textual cues embedded in sign translation texts. These limitations inherently diminish the representative capacity of pre-trained models. To this end, we present a multimodal SLP framework to leverage rich visual contextual information and vision-language semantic consistency with massively available data to enhance the representa…
-
TransSLR: A Lightweight Transformer for Sign Language RecognitionWanchi Lucia Yen, S. Johnny, V. Olufemi et al. · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet
Abstract
Automated sign language recognition for underrepresented languages remains a largely unsolved problem. Central African Sign Language (CASL) exemplifies this challenge: the CASL-W60 benchmark contains 60 isolated signs, and the best published result is 69.93%. We present TransSLR, a lightweight Temporal Transformer Encoder trained from scratch on normalized 64-frame pose sequences. By operating on geometric keypoint representations rather than raw RGB, TransSLR achieves 80.39% top-1 accuracy and 91.07% top-5 accuracy on a signer-independent evaluation split. This result is 10.46 percentage points higher than the published 69.93% result. TransSLR has 8.67 million trainable parameters. We also …
-
SignVLM: a pre-trained large video model for sign language recognitionHamzah Luqman · 2025 · PeerJ Computer Science · cited 7 times · Not summarized yet
Abstract
Sign language recognition (SLR) plays a vital role in including people with hearing impairment in the community. It facilitates the recognition of sign gestures and converts them into spoken languages. One of the main challenges for developing SLR systems is the lack of annotated datasets. This issue is more noticeable with low–resourced sign languages. To address this issue, we propose a pretrained large vision model, SignVLM, for SLR. This work explores the capability of the contrastive language–image pre-training (CLIP) model for SLR. This model is used to extract spatial features from the sign video frames while a Transformer decoder is used for temporal learning. The proposed model has …
Prior work it builds on
-
Hand-Model-Aware Sign Language RecognitionHe-Zhen Hu, Wen-Gang Zhou, Hou-Qiang Li · 2021 · AAAI Conference on Artificial Intelligence · cited 100 times · Not summarized yet
Abstract
Hand gestures play a dominant role in the expression of sign language. Current deep-learning based video sign language recognition (SLR) methods usually follow a data-driven paradigm under the supervision of the category label. However, those methods suffer limited interpretability and may encounter the overfitting issue due to limited sign data sources. In this paper, we introduce the hand prior and propose a new hand-model-aware framework for isolated SLR with the modeling hand as the intermediate representation. We first transform the cropped hand sequence into the latent semantic feature. Then the hand model introduces the hand prior and provides a mapping from the semantic feature to th…
-
Global-Local Enhancement Network for NMF-Aware Sign Language RecognitionHe-Zhen Hu, Wen-Gang Zhou, Junfu Pu et al. · 2020 · ACM Trans. Multim. Comput. Commun. Appl. · cited 86 times · Not summarized yet
Abstract
Sign language recognition (SLR) is a challenging problem, involving complex manual features (i.e., hand gestures) and fine-grained non-manual features (NMFs) (i.e., facial expression, mouth shapes, etc.). Although manual features are dominant, non-manual features also play an important role in the expression of a sign word. Specifically, many sign words convey different meanings due to non-manual features, even though they share the same hand gestures. This ambiguity introduces great challenges in the recognition of sign words. To tackle the above issue, we propose a simple yet effective architecture called Global-Local Enhancement Network (GLE-Net), including two mutually promoted streams t…
-
BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuesSamuel Albanie, Gül Varol, Liliane Momeni et al. · 2020 · European Conference on Computer Vision · cited 248 times · Not summarized yet
Abstract
Recent progress in fine-grained gesture and action classification, and machine translation, point to the possibility of automated sign language recognition becoming a reality. A key stumbling block in making progress towards this goal is a lack of appropriate training data, stemming from the high complexity of sign annotation and a limited supply of qualified annotators. In this work, we introduce a new scalable approach to data collection for sign recognition in continuous videos. We make use of weakly-aligned subtitles for broadcast footage together with a keyword spotting method to automatically localise sign-instances for a vocabulary of 1,000 signs in 1,000 hours of video. We make the f…
-
Attention-Based 3D-CNNs for Large-Vocabulary Sign Language RecognitionJie Huang, Wen-Gang Zhou, Hou-Qiang Li et al. · 2019 · IEEE transactions on circuits and systems for video technology (Print) · cited 201 times · Not summarized yet
Abstract
Sign language recognition (SLR) is an important and challenging research topic in the multimedia field. Conventional techniques for SLR rely on hand-crafted features, which achieve limited success. In this paper, we present attention-based 3D-convolutional neural networks (3D-CNNs) for SLR. The framework has two advantages: 3D-CNNs learn spatio-temporal features from raw video without prior knowledge and the attention mechanism helps to select the clue. When training 3D-CNN for capturing spatio-temporal features, spatial attention is incorporated into the network to focus on the areas of interest. After feature extraction, temporal attention is utilized to select the significant motions for …
-
VL-BERT: Pre-training of Generic Visual-Linguistic RepresentationsWeijie Su, Xi-Zhou Zhu, Yue Cao et al. · 2019 · International Conference on Learning Representations · cited 1,867 times · Not summarized yet
Abstract
We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-trai…
Papers that cite it
-
MASA: Motion-Aware Masked Autoencoder With Semantic Alignment for Sign Language RecognitionWei-Chao Zhao, He-Zhen Hu, Wen-Gang Zhou et al. · 2024 · IEEE transactions on circuits and systems for video technology (Print) · cited 32 times · Not summarized yet
Abstract
Sign language recognition (SLR) has long been plagued by insufficient model representation capabilities. Although current pre-training approaches have alleviated this dilemma to some extent and yielded promising performance by employing various pretext tasks on sign pose data, these methods still suffer from two primary limitations: i) Explicit motion information is usually disregarded in previous pretext tasks, leading to partial information loss and limited representation capability. ii) Previous methods focus on the local context of a sign pose sequence, without incorporating the guidance of the global meaning of lexical signs. To this end, we propose a Motion-Aware masked autoencoder wit…
-
SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language RecognitionKun Xie, Zhixi Cai, Kalin Stefanov · 2026 · International Conference on Pattern Recognition · Not summarized yet
Abstract
Subtle hand differences make sign language recognition challenging, yet many existing methods rely on encoders pretrained on generic action datasets that poorly capture such fine-grained cues. We propose a self-supervised pretraining method for sign language recognition that uses segmentation-based masking to adapt to the presence and motion of key body parts, rather than treating hand poses as static visual tokens. The resulting mask-and-reconstruct objective improves fine-grained sign representation learning. On WLASL, NMFs-CSL, and Slovo, our encoder achieves state-of-the-art performance, improving per-instance and per-class Top-1 accuracy while using fewer input frames and modalities tha…
-
Self-Supervised Representation Learning With Spatial-Temporal Consistency for Sign Language RecognitionWei-Chao Zhao, Wen-Gang Zhou, He-Zhen Hu et al. · 2024 · IEEE Transactions on Image Processing · cited 20 times · Not summarized yet
Abstract
Recently, there have been efforts to improve the performance in sign language recognition by designing self-supervised learning methods. However, these methods capture limited information from sign pose data in a frame-wise learning manner, leading to sub-optimal solutions. To this end, we propose a simple yet effective self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency from two distinct perspectives and learn instance discriminative representation for sign language recognition. On one hand, since the semantics of sign language are expressed by the cooperation of fine-grained hands and coarse-grained trunks, we utilize both granularity in…
-
SignBart - New approach with the skeleton sequence for Isolated Sign language RecognitionT. Nguyen, Minh Khue Phan Tran · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
Abstract
Sign language recognition is crucial for individuals with hearing impairments to break communication barriers. However, previous approaches have had to choose between efficiency and accuracy. Such as RNNs, LSTMs, and GCNs, had problems with vanishing gradients and high computational costs. Despite improving performance, transformer-based methods were not commonly used. This study presents a new novel SLR approach that overcomes the challenge of independently extracting meaningful information from the x and y coordinates of skeleton sequences, which traditional models often treat as inseparable. By utilizing an encoder-decoder of BART architecture, the model independently encodes the x and y …
-
StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language RecognitionXiao-Long Shen, Zhedong Zheng, Yi Yang · 2022 · ACM Trans. Multim. Comput. Commun. Appl. · cited 33 times · Not summarized yet
Abstract
The goal of sign language recognition (SLR) is to help those who are hard of hearing or deaf overcome the communication barrier. Most existing approaches can be typically divided into two lines, i.e., Skeleton-based, and RGB-based methods, but both lines of methods have their limitations. Skeleton-based methods do not consider facial expressions, while RGB-based approaches usually ignore the fine-grained hand structure. To overcome both limitations, we propose a new framework called the Spatial-temporal Part-aware network (StepNet), based on RGB parts. As its name suggests, it is made up of two modules: Part-level Spatial Modeling and Part-level Temporal Modeling. Part-level Spatial Modeling…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 5 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.