# Text Embeddings by Weakly-Supervised Contrastive Pre-training

**Authors:** Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei  
**arXiv:** [2212.03533](https://arxiv.org/abs/2212.03533)  
**Date:** 2022-12-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.03533v1/

## Key points

- E5 is a state-of-the-art text embedding model
- Trained using weak supervision signals from CCPairs dataset
- Can be used for retrieval, clustering, and classification tasks
- Impressive performance in zero-shot and fine-tuned settings
- Extensive evaluations on 56 datasets from BEIR and MTEB benchmarks
- Historical context on text embeddings, mentioning LSA and LDA
- Weighted average of word vectors as a baseline for sentence embeddings
- Pre-trained language models and labeled datasets like SNLI and MS-MARCO used for fine-tuning
- Contrastive loss found to be more effective than classification-based losses for embeddings
- Models extend contrastive loss to multilingual and multi-modal scenarios
- Self-supervised pre-training tasks for text matching and retrieval discussed
- Evaluation of text embeddings challenging, with benchmarks measuring downstream task performances
- Relevance to community efforts by sentence-transformers in training embeddings with labeled and automatically collected datasets highlighted
- High-quality embeddings can be trained using self-supervised pre-training only

## Summary

This paper introduces E5, a family of state-of-the-art text embeddings that excel in various tasks. The model is trained using weak supervision signals from a large-scale text pair dataset called CCPairs. E5 can be used as a general-purpose embedding model for tasks like retrieval, clustering, and classification. It achieves impressive performance in both zero-shot and fine-tuned settings. The authors conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. The paper provides some historical context on text embeddings, mentioning early works such as Latent Semantic Indexing (LSA) and Latent Dirichlet Allocation (LDA). It also highlights the effectiveness of simple weighted average of word vectors as a baseline for sentence embeddings. With the development of pre-trained language models and labeled datasets like SNLI and MS-MARCO, methods like Sentence-BERT, SimCSE, Sentence-T5, and SGPT directly fine-tune language models to output continuous embeddings. Contrastive loss has been found to be more effective than classification-based losses for embeddings. Several models extend contrastive loss to multilingual and multi-modal scenarios using parallel sentences and image-text pairs. Another direction is self-supervised pre-training tasks for text matching and retrieval. The paper discusses previous approaches that use synthetic data for training but struggle to match the performance of BM25 without further fine-tuning on labeled datasets. Evaluation and interpretation of text embeddings are challenging, with benchmarks measuring embedding quality through downstream task performances. The authors highlight their work's relevance to community efforts by sentence-transformers in training embeddings with labeled and automatically collected datasets. They demonstrate that high-quality embeddings can be trained using self-supervised pre-training only. In conclusion, this work presents E5 as a powerful text embedding model trained with weak supervision signals. It achieves strong performance across various tasks when fine-tuned on less labeled data compared to other models.