# BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

**Authors:** Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, Zheng Liu  
**arXiv:** [2402.03216](https://arxiv.org/abs/2402.03216)  
**Date:** 2024-02-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.03216v1/

## Key points

- M3-Embedding is a versatile model that supports over 100 languages and excels in multi-lingual and cross-lingual retrieval tasks.
- It can handle dense, multi-vector, and sparse retrieval simultaneously, making it valuable for real-world information retrieval applications.
- The model can process inputs from short sentences to long documents with up to 8192 tokens.
- A novel self-knowledge distillation approach using relevance scores from different retrieval functionalities enhances the training quality of M3-Embedding.
- Optimized batching strategy enables large batch sizes and high training throughput for more discriminative embeddings.
- In experiments evaluating multilingual retrieval, cross-lingual retrieval, and long-document retrieval tasks, M3-Embedding consistently demonstrates superior performance across various languages and input lengths.
- Even without fine-tuning on long document data, M3-Embedding outperforms most baselines due to its robust pre-training stage.
- A simple strategy called MCLS addresses situations where fine-tuning for document retrieval is not feasible or resource-intensive.

## Summary

, , , , 

Introducing M3-Embedding: A Versatile Model for Multilingual, Multi-functional, and Multi-granular Embeddings

M3-Embedding is a powerful embedding model that supports over 100 languages and achieves state-of-the-art performance in multi-lingual and cross-lingual retrieval tasks. It excels in handling dense, multi-vector, and sparse retrieval simultaneously, making it a valuable tool for real-world information retrieval applications. Notably, M3-Embedding can process inputs ranging from short sentences to long documents with up to 8192 tokens. To further enhance the training quality of M3-Embedding, we propose a novel self-knowledge distillation approach where relevance scores from different retrieval functionalities serve as teacher signals. We also optimize the batching strategy to enable large batch sizes and high training throughput for more discriminative embeddings. This technical innovation sets M3-Embedding apart as the first model with such strong versatility. In our experiments evaluating multilingual retrieval, cross-lingual retrieval, and long-document retrieval tasks, M3-Embedding consistently demonstrates superior performance across various languages and input lengths. To showcase its proficiency in handling long inputs, we evaluate its performance on benchmarks like MLDR (Multilingual Long-Doc Retrieval) curated from multilingual articles on Wikipedia, Wudao, mC4, and NarrativeQA. Ablation studies show that even without fine-tuning on long document data (Dense-w.o.long), M3-Embedding outperforms most baselines due to its robust pre-training stage. Additionally, we introduce a simple strategy called MCLS to address situations where fine-tuning for document retrieval is not feasible or resource-intensive. Our analysis on NarrativeQA demonstrates that as sequence length increases, M3-Embedding consistently outperforms baselines in long-document retrieval tasks. Furthermore, we conduct experiments on self-knowledge distillation and multi-stage training with MIRACL (nDCG@10), showcasing the model's proficiency in handling long inputs. We also present an MCLS strategy to enhance the model's long-text capabilities without requiring additional training resources.