# data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

**Authors:** Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli  
**arXiv:** [2202.03555](https://arxiv.org/abs/2202.03555)  
**Date:** 2022-02-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2202.03555v1/

## Key points

- Self-supervised learning is widely used across different modalities
- Algorithms and objectives vary depending on the specific modality
- A new framework called data2vec has been introduced to address this issue
- Data2vec uses the same learning method for speech, NLP, and computer vision
- The approach involves predicting latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture
- Data2vec predicts contextualized latent representations that contain information from the entire input, unlike traditional approaches that predict modality-specific targets such as words or visual tokens
- The framework has demonstrated state-of-the-art or competitive performance compared to existing approaches in major benchmarks for speech recognition, image classification, and natural language understanding
- In NLP tasks specifically, data2vec outperforms RoBERTa baseline when masking spans of four BPE tokens with masking probability 0.35.
- Data2vec does not leave tokens unmasked or use random targets as in BERT models.
- The framework allows for an open vocabulary setting where new target types can be defined by the model as needed.
- Layer-averaged targets have been used in data2vec to improve performance compared to BYOL methods in computer vision.
- Data2vec presents a promising step towards general self-supervised learning across different modalities and shows potential for further advancements in these fields.

## Summary

The concept of self-supervised learning is widely used across different modalities, but the algorithms and objectives vary depending on the specific modality. To address this issue, a new framework called data2vec has been introduced that uses the same learning method for speech, natural language processing (NLP), and computer vision. The approach involves predicting latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture. Unlike traditional approaches that predict modality-specific targets such as words or visual tokens, data2vec predicts contextualized latent representations that contain information from the entire input. The framework has been tested on major benchmarks for speech recognition, image classification, and natural language understanding and has demonstrated state-of-the-art or competitive performance compared to existing approaches. In NLP tasks specifically, data2vec outperforms RoBERTa baseline when masking spans of four BPE tokens with masking probability 0.35. This approach does not leave tokens unmasked or use random targets as in BERT models. Instead, it predicts contextualized latent representations emerging from self-attention over the entire unmasked text sequence without relying on discrete units like words or subwords as training targets. The framework also allows for an open vocabulary setting where new target types can be defined by the model as needed. Additionally, layer-averaged targets have been used in data2vec to improve performance compared to BYOL methods in computer vision. Overall, data2vec presents a promising step towards general self-supervised learning across different modalities and shows potential for further advancements in these fields.