# SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval

**Authors:** Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant  
**arXiv:** [2109.10086](https://arxiv.org/abs/2109.10086)  
**Date:** 2021-09-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2109.10086v1/

## Key points

- Significant focus on enhancing the initial retriever in ranking pipelines in Neural Information Retrieval (IR)
- Growing interest in exploring sparse representations for documents and queries to leverage advantages of traditional bag-of-words models
- Introduction of SPLADE model providing highly sparse representations with competitive performance
- Proposed enhancements to SPLADE including modifications to pooling mechanism, benchmarking based on document expansion, and distillation techniques
- Exploration of dense retrieval approaches based on BERT Siamese models for candidate generation in Question Answering and IR tasks
- Alternative approaches like SNRM embedding documents and queries into a sparse latent space using ℓ1 regularization
- Attempts to transfer knowledge from pre-trained language models like BERT to sparse approaches such as DeepCT
- Utilization of document expansion through generative methods to address vocabulary mismatch challenges

## Summary

, , , , 

In the realm of Neural Information Retrieval (IR), there has been a significant focus on enhancing the initial retriever in ranking pipelines. One effective approach involves learning dense embeddings for retrieval using efficient approximate nearest neighbors methods, which has shown promising results. However, there is also a growing interest in exploring sparse representations for documents and queries, aiming to leverage the advantages of traditional bag-of-words models such as exact term matching and the efficiency of inverted indexes. Recently, the SPLADE model was introduced as a solution that provides highly sparse representations while delivering competitive performance compared to state-of-the-art dense and sparse approaches. Building upon SPLADE, this paper proposes several substantial improvements in terms of effectiveness and efficiency. These enhancements include modifications to the pooling mechanism, benchmarking a model based solely on document expansion, and introducing models trained with distillation techniques. The results obtained from these enhancements are reported on the BEIR benchmark. Furthermore, related works in the field have explored various approaches to dense retrieval based on BERT Siamese models, which have become standard for candidate generation in Question Answering and IR tasks. Recent studies emphasize the importance of training strategies to achieve optimal results with these models. Additionally, alternative approaches such as term-based indexes have been explored, with models like SNRM aiming to embed documents and queries into a sparse latent space using ℓ1 regularization. Moreover, inspired by the success of BERT, attempts have been made to transfer knowledge from pre-trained language models to sparse approaches like DeepCT. This approach focuses on learning contextualized term weights within the full vocabulary space but faces challenges related to vocabulary mismatch. To address this issue, techniques like document expansion through generative methods have been employed. Overall, by combining insights from existing research and introducing novel improvements to sparse retrieval models like SPLADE, this study contributes significantly towards advancing the effectiveness and efficiency of neural Information Retrieval systems.