# Improving Text Embeddings with Large Language Models

**Authors:** Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei  
**arXiv:** [2401.00368](https://arxiv.org/abs/2401.00368)  
**Date:** 2023-12-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.00368v1/

## Key points

- Introduces a novel method for obtaining high-quality text embeddings using synthetic data and few training steps
- Does not require complex training pipelines or manually collected datasets
- Leverages proprietary Language Model Models (LLMs) to generate diverse synthetic data for text embedding tasks in multiple languages
- Fine-tunes open-source decoder-only LLMs on synthetic data using standard contrastive loss
- Achieves strong performance on competitive text embedding benchmarks without using labeled data
- Sets new state-of-the-art results on BEIR and MTEB benchmarks when fine-tuned with a mixture of synthetic and labeled data
- Discusses future work, including improving multilingual performance and exploring the use of open-source LLMs for synthetic data generation
- Aims to improve inference efficiency and reduce storage costs for LLM-based text embeddings
- Provides statistics on generated synthetic data: 500k examples in 93 languages, generated by GPT-4 and GPT-35-Turbo (with acceptable quality)
- Trained model evaluated on MTEB benchmark, achieving highest average score and outperforming previous state-of-the-art models by 2.4 points
- Builds upon previous work by leveraging LLMs and synthetic data to enhance text embeddings
- Demonstrates that high-quality text embeddings can be obtained using synthetic data and streamlined training process with LLMs

## Summary

This paper introduces a novel and simple method for obtaining high-quality text embeddings using only synthetic data and a small number of training steps. Unlike existing methods that rely on multi-stage pre-training with weakly-supervised text pairs, followed by fine-tuning with labeled datasets, this method does not require complex training pipelines or manually collected datasets. Instead, proprietary Language Model Models (LLMs) are leveraged to generate diverse synthetic data for text embedding tasks in multiple languages. Open-source decoder-only LLMs are then fine-tuned on the synthetic data using standard contrastive loss. Experimental results show that this method achieves strong performance on competitive text embedding benchmarks without using any labeled data. When fine-tuned with a mixture of synthetic and labeled data, the model sets new state-of-the-art results on the BEIR and MTEB benchmarks. The paper also discusses future work, including improving multilingual performance and exploring the use of open-source LLMs to generate synthetic data. Additionally, efforts will be made to improve inference efficiency and reduce storage costs for LLM-based text embeddings. The paper provides statistics on the generated synthetic data, which includes 500k examples with instructions in 93 languages. The majority of examples are generated by GPT-4, with some generated by GPT-35-Turbo. While some outputs from GPT-35-Turbo do not strictly follow prompt guidelines, the overall quality is acceptable. The model is fine-tuned using a combination of the generated synthetic data and public datasets. The trained model is evaluated on the MTEB benchmark, achieving the highest average score and outperforming previous state-of-the-art models by 2.4 points. In related work, text embeddings have been extensively studied as continuous low-dimensional representations of text. This paper builds upon previous work by leveraging LLMs and synthetic data to enhance text embeddings. Overall, this paper demonstrates that high-quality text embeddings can be obtained using synthetic data and a streamlined training process with LLMs. The method achieves strong performance on text embedding benchmarks and sets new state-of-the-art results when combined with labeled data. Future work will focus on further improving multilingual performance, exploring open-source LLMs for synthetic data generation, and optimizing inference efficiency and storage costs.