# LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

**Authors:** Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, Siva Reddy  
**arXiv:** [2404.05961](https://arxiv.org/abs/2404.05961)  
**Date:** 2024-04-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.05961v1/

## Key points

- Text embedding models are crucial in NLP for encoding semantic content into vector representations
- Bidirectional encoders like BERT and T5 have traditionally been used for text embedding tasks
- Decoder-only large language models (LLMs) are emerging as powerful alternatives for text embedding tasks
- LLM2Vec enables bidirectional attention, masked next token prediction, and unsupervised contrastive learning to transform decoder-only LLMs into robust text encoders
- Significant advancements have been made in English word- and sequence-level tasks using LLM2Vec with parameters ranging from 1.3B to 7B
- LLM2Vec outperforms traditional encoder-only models on word-level tasks and achieves new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB)
- Combining LLM2Vec with supervised contrastive learning techniques leads to state-of-the-art results on MTEB among models trained solely on publicly available data
- The study by Parishad BehnamGhader et al. showcases the efficacy of transforming large language models into powerful text encoders without costly adaptations or synthetic data generation

## Summary

In the field of Natural Language Processing (NLP), text embedding models play a crucial role in encoding the semantic content of text into vector representations. This enables various NLP tasks such as semantic textual similarity and information retrieval. Traditionally, bidirectional encoders like BERT and T5 have been widely used for text embedding tasks. However, a recent shift in the community has seen the emergence of decoder-only large language models (LLMs) as powerful alternatives. are becoming popular for text embedding tasks due to their ability to transform any decoder-only LLM into a robust text encoder. This is made possible through  a novel approach that involves three key steps: enabling bidirectional attention, masked next token prediction, and unsupervised contrastive learning. By applying LLM2Vec to popular LLMs with parameters ranging from 1.3B to 7B, significant advancements have been made in English word- and sequence-level tasks. Notably,  using LLM2Vec outperform traditional encoder-only models by a considerable margin on word-level tasks and achieve a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB). Furthermore, when combining LLM2Vec with supervised contrastive learning techniques, state-of-the-art results are achieved on MTEB among models trained solely on publicly available data. These empirical findings underscore the efficacy of transforming  in a parameter-efficient manner without resorting to costly adaptations or synthetic data generation. The study conducted by Parishad BehnamGhader et al., showcases how  of large language models as powerful text encoders. This innovative approach not only enhances performance on existing benchmarks but also paves the way for future advancements in natural language understanding and processing tasks.