# Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs

**Authors:** Yuchen Fu, Zifeng Cheng, Zhiwei Jiang, Zhonghui Wang, Yafeng Yin, Zhengliang Li, Qing Gu  
**arXiv:** [2412.11556](https://arxiv.org/abs/2412.11556)  
**Date:** 2024-12-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.11556v1/

## Key points

- Authors explore extracting sentence embeddings from large language models (LLMs)
- Highlight the potential of LLMs in showcasing robust semantic understanding capabilities
- Introduce Token Prepending (TP) technique to address limitations in decoder-only LLMs with causal attention
- TP involves adding each layer's decoded sentence embedding at the beginning of the input sentence for the next layer
- TP is described as plug-and-play and training-free, allowing seamless integration with various prompt-based methods and autoregressive LLMs
- Extensive experiments validate effectiveness of TP on Semantic Textual Similarity (STS) tasks and downstream classification tasks
- Significant performance improvements observed across different LLMs compared to existing prompt-based methods
- Enhancements achieved with minimal additional inference cost

## Summary

, , , , 

In their paper titled "Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs," authors Yuchen Fu, Zifeng Cheng, Zhiwei Jiang, Zhonghui Wang, Yafeng Yin, Zhengliang Li, and Qing Gu delve into the realm of extracting sentence embeddings from large language models (LLMs). They highlight the potential of LLMs in showcasing robust semantic understanding capabilities. Previous research has primarily focused on prompt engineering to extract sentence embeddings by encoding information into the last token's embedding. However, the authors point out a limitation in decoder-only LLMs with causal attention that restricts earlier tokens in a sentence from attending to later tokens. This can lead to biased encoding and subsequent effects on the final decoded token. To address this issue, the authors introduce a novel technique called Token Prepending (TP). This approach involves adding each layer's decoded sentence embedding at the beginning of the input sentence for the next layer. By doing so, earlier tokens can now access complete sentence information under the causal attention mechanism. The TP technique is described as plug-and-play and training-free, allowing seamless integration with various prompt-based sentence embedding methods and autoregressive LLMs. The effectiveness of the proposed TP technique is validated through extensive experiments on Semantic Textual Similarity (STS) tasks and downstream classification tasks. The results demonstrate significant performance improvements across different LLMs when compared to existing prompt-based methods. Importantly, these enhancements are achieved with minimal additional inference cost. Overall, this study sheds light on a promising approach for enhancing sentence embeddings from LLMs by addressing limitations in existing methodologies. The Token Prepending technique offers a practical and efficient solution that could potentially advance various natural language processing applications requiring robust semantic understanding capabilities.