# ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

**Authors:** Omar Khattab, Matei Zaharia  
**arXiv:** [2004.12832](https://arxiv.org/abs/2004.12832)  
**Date:** 2020-04-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2004.12832v1/

## Key points

- Advancements in NLU have impacted IR, particularly in document ranking
- Deep LMs like BERT improve accuracy but come with high computational costs
- ColBERT is a novel ranking model designed for efficient retrieval using deep LMs
- ColBERT's late interaction architecture optimizes efficiency by deferring detailed interactions
- ColBERT accelerates query processing speed and enables pre-computation of document representations offline
- ColBERT's interaction mechanism allows for pruning and leveraging vector-similarity indexes for faster retrieval
- Extensive evaluations show that ColBERT matches existing BERT-based models and outperforms non-BERT baselines
- ColBERT executes queries much faster and requires significantly fewer FLOPs per query compared to other models

## Summary

, , , , 

In recent years, advancements in NLU have significantly impacted IR, specifically in the realm of document ranking. The utilization of deep LMs has proven to be highly effective in improving the accuracy of ranking models. However, a notable drawback of these models is the substantial increase in computational cost compared to previous methods. This heightened cost is primarily due to the necessity of processing each query-document pair through a large neural network to calculate a single relevance score. To address this challenge, a novel ranking model called ColBERT has been introduced. ColBERT is designed to optimize the efficiency of retrieval by adapting deep LMs, particularly BERT, for more streamlined performance. One key feature of ColBERT is its late interaction architecture, which involves independently encoding both the query and document using BERT before implementing a cost-effective yet robust interaction step that captures their fine-grained similarity. By deferring but still preserving this detailed interaction process, ColBERT can leverage the expressive capabilities of deep LMs while also enabling the pre-computation of document representations offline. This innovative approach significantly accelerates query processing speed. Moreover, beyond simply reducing re-ranking costs associated with traditional models, ColBERT's interaction mechanism is conducive to pruning and facilitates leveraging vector-similarity indexes for end-to-end retrieval directly from extensive document collections. Extensive evaluations conducted on two recent passage search datasets have demonstrated that ColBERT's effectiveness is on par with existing BERT-based models and surpasses all non-BERT baselines. Impressively, ColBERT executes queries at a rate two orders-of-magnitude faster and requires four orders-of-magnitude fewer FLOPs per query compared to other models. Authored by Omar Khattab and Matei Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT" presents a groundbreaking approach that not only enhances retrieval efficiency but also showcases remarkable performance improvements in information retrieval tasks within NLU frameworks. Accepted at SIGIR 2020, this research signifies a significant advancement in optimizing deep language models for efficient document ranking processes.