# Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers

**Authors:** Shijie Chen, Bernal Jiménez Gutiérrez, Yu Su  
**arXiv:** [2410.02642](https://arxiv.org/abs/2410.02642)  
**Date:** 2024-10-03  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2410.02642v1/

## Key points

- Large language models (LLMs) are increasingly important in information retrieval systems, especially in generative AI and retrieval-augmented generation.
- Existing LLM-based re-ranking methods heavily rely on autoregressive generation, limiting their applicability to specialized or proprietary models.
- In-context re-ranking (ICR) is a novel approach that leverages changes in attention patterns induced by search queries for accurate and efficient re-ranking without autoregressive generation.
- ICR requires only two forward passes to re-rank N documents compared to generative methods that demand at least O(N) forward passes, making it more efficient.
- ICR can be applied to any LLM without specialized training and ensures a well-formed ranking output.
- Experiments show that ICR outperforms RankGPT in performance while reducing latency by over 60% in practical scenarios, particularly excelling in tasks requiring complex re-ranking signals.

## Summary

, , , , 

In the realm of information retrieval (IR) systems, the role of large language models (LLMs) has become increasingly prominent. This is especially true in the era of generative AI and retrieval-augmented generation. These LLMs boast powerful language processing capabilities and remarkable versatility, making them popular choices for zero-shot re-ranking within IR systems. However, existing LLM-based re-ranking methods heavily rely on autoregressive generation, which limits their applicability to specialized or proprietary models. This raises the question: is autoregressive generation truly necessary and optimal for LLMs to excel in re-ranking tasks? To address this query, a novel approach called in-context re-ranking (ICR) is proposed. Unlike traditional generative methods, ICR leverages the change in attention patterns induced by search queries to achieve accurate and efficient re-ranking without relying on autoregressive generation. Additionally, a calibration method using content-free queries is introduced to mitigate intrinsic biases in LLMs. One key advantage of ICR is its efficiency - requiring only two ($O(1)$) forward passes to re-rank $N$ documents compared to generative methods that demand at least $O(N)$ forward passes. Moreover, ICR can be seamlessly applied to any LLM without specialized training while ensuring a well-formed ranking output. Extensive experiments conducted with two popular open-weight LLMs on standard single-hop and multi-hop information retrieval benchmarks demonstrate that ICR surpasses RankGPT in performance while reducing latency by over 60% in practical scenarios. Detailed analyses further reveal that ICR excels particularly in tasks necessitating complex re-ranking signals. These findings underscore the potential for exploring innovative ways of harnessing open-weight LLMs beyond text generation applications. The development of ICR represents a significant step towards maximizing the utility of LLMs in information retrieval tasks by leveraging inherent signals within these models more effectively than traditional generative approaches.