# LLMs are Also Effective Embedding Models: An In-depth Overview

**Authors:** Chongyang Tao, Tao Shen, Shen Gao, Junshuo Zhang, Zhen Li, Zhengwei Tao, Shuai Ma  
**arXiv:** [2412.12591](https://arxiv.org/abs/2412.12591)  
**Date:** 2024-12-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.12591v1/

## Key points

- Large language models (LLMs) have revolutionized natural language processing with exceptional performance across tasks.
- Shift towards decoder-only LLMs like GPT, LLaMA, and Mistral for embedding models.
- Two primary strategies for deriving embeddings: direct prompting and data-centric tuning.
- Direct prompting involves prompt designs and rationale for competitive embeddings.
- Data-centric tuning covers model architecture, training objectives, and data constructions influencing embedding tuning.
- Challenges addressed include handling longer texts, multilingual/cross-modal data, performance/efficiency trade-offs, dense vs sparse embeddings, pooling strategies, scaling laws.
- Adaptation challenges of LLMs for embeddings: cross-task quality issues, efficiency vs accuracy trade-offs, low-resource scenarios, long-context considerations, data biases, robustness concerns.
- Extension of context in LLM embedding models through plug-and-play methods to address high computational costs associated with fine-tuning LLMs (e.g., chunking-free architecture proposed by Luo et al.).
- Survey serves as a valuable resource synthesizing advancements in LLMs as embedding models and highlights key challenges for future work in enhancing their effectiveness and efficiency within NLP tasks.

## Summary

Large language models (LLMs) have transformed natural language processing by achieving exceptional performance across various tasks. Recently, there has been a shift towards recognizing their effectiveness as embedding models, moving away from traditional encoder-only models like ELMo and BERT to decoder-only, large-scale LLMs such as GPT, LLaMA, and Mistral. This comprehensive survey delves into this transition and explores two primary strategies for deriving embeddings: direct prompting and data-centric tuning. were first discussed before diving into the details of LLM-based embedding models. Direct prompting involves discussing prompt designs and the rationale behind generating competitive embeddings. On the other hand, data-centric tuning covers various aspects influencing the tuning of an embedding model, including model architecture, training objectives, and data constructions. Additionally,, such as handling longer texts and multilingual/cross-modal data are explored in detail. The survey also examines factors impacting the selection of embedding models such as performance/efficiency trade-offs, dense vs sparse embeddings, pooling strategies, and scaling laws. Furthermore,in adapting LLMs for embeddings are highlighted. These include cross-task embedding quality issues, balancing efficiency with accuracy trade-offs, low-resource scenarios, long-context considerations, data biases,and robustness concerns. Moreover,have focused on extending the context of LLM embedding models through plug-and-play methods to address high computational costs associated with fine-tuning LLMs. For instance,Luo et al. (2024a) proposed a chunking-free architecture to handle long contexts effectively. In conclusion,this survey serves as a valuable resource for researchers and practitioners by synthesizing current advancements in LLMs as embedding models while shedding light on key challenges. It offers a comprehensive framework for future work aimed at enhancing the effectiveness and efficiency of LLMs in their role as embedding models within natural language processing tasks.