# Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models

**Authors:** Michael Günther, Isabelle Mohr, Bo Wang, Han Xiao  
**arXiv:** [2409.04701](https://arxiv.org/abs/2409.04701)  
**Date:** 2024-09-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.04701v1/

## Key points

- In text retrieval, it is important to extract smaller segments for various use cases
- Traditional chunking methods may result in a loss of contextual information from surrounding chunks
- Late chunking method utilizes advanced open-source models to embed all tokens of a long text before applying chunking just before mean pooling
- This approach ensures that chunk embeddings capture full contextual information, leading to superior results without additional training
- Late chunking can be seamlessly integrated into any long-context text embedding model and does not require extra training
- The code for this method is available on GitHub for reproducibility
- Late chunking offers a promising solution to enhance text retrieval by preserving contextual information within chunk embeddings

## Summary

, , , , 

In the realm of text retrieval, it is crucial to extract smaller segments of text for various use cases. This is where  excel, as they perform better with shorter text segments due to less "over-compression" in the embeddings. To achieve this, practitioners often split text documents into smaller chunks and encode them separately. However, this can result in a loss of contextual information from surrounding chunks, leading to suboptimal representations. To address this issue, a novel approach known as  has been introduced. This method utilizes advanced open-source models such as jina-embeddings-v2 to first embed all tokens of a long text before applying chunking just before mean pooling. By doing so, the resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks without requiring additional training. Furthermore, this method is versatile and can be applied to any . The limitations of traditional chunking methods are highlighted through an illustration using a Wikipedia article on Berlin that is split into chunks. It becomes evident that phrases like "its" and "the city," which reference "Berlin," are mentioned only in the first sentence, making it challenging for the embedding model to link these references accurately. Late chunking overcomes these limitations by utilizing the capabilities of advanced open-source models such as jina-embeddings-v2 to encode all tokens of documents before applying chunking just before mean pooling. This approach ensures that chunk embeddings retain crucial contextual information from the entire text, leading to improved performance compared to conventional chunking methods across various retrieval benchmarks. Notably, late chunking does not require additional training and can be seamlessly integrated into any long-context text embedding model. To facilitate reproducibility, the code for this method has been made available on GitHub. The paper delves into related work in Section 2, explains the late chunking method in Section 3, presents an evaluation in Section 4 showcasing its superiority over traditional approaches, and concludes with insights in Section 5. In conclusion,  offers a promising solution to enhance text retrieval by preserving contextual information within chunk embeddings derived from long texts. Its effectiveness and versatility make it a valuable addition to existing techniques for optimizing retrieval tasks across different domains and applications.