# Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

**Authors:** Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, Michael Bendersky  
**arXiv:** [2306.17563](https://arxiv.org/abs/2306.17563)  
**Date:** 2023-06-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.17563v1/

## Key points

- Large Language Models (LLMs) have limited success in text ranking compared to baseline rankers
- Existing methods struggle to outperform baseline rankers, except for a recent approach using a blackbox commercial system
- The authors propose Pairwise Ranking Prompting (PRP) as a new technique to address this issue
- LLMs do not fully understand pointwise and listwise ranking prompts due to their training process
- PRP reduces the burden on LLMs by introducing a new approach
- PRP achieves state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs
- PRP outperforms previous approaches based on larger models by over 5% at NDCG@1 and other existing solutions by over 10% for nearly all ranking metrics
- Several variants of PRP are proposed to improve efficiency with competitive results even with linear complexity
- PRP has additional benefits such as supporting both generation and scoring LLM APIs and being insensitive to input ordering.

## Summary

Large Language Models (LLMs) have shown impressive performance on various natural language tasks. However, they have limited success in the text ranking problem compared to well-trained baseline rankers. Existing methods struggle to outperform these baseline rankers, and the only exception is a recent approach that relies on a blackbox commercial system. In this paper, the authors propose a new technique called Pairwise Ranking Prompting (PRP) to address this issue. The authors analyze the pointwise and listwise ranking prompts used by existing methods and argue that LLMs do not fully understand these ranking formulations due to their training process. Pointwise approaches require LLMs to output calibrated prediction probabilities before sorting, which is challenging for them. Listwise approaches also generate conflicting or useless outputs on moderate-sized LLMs. To overcome these limitations, the authors introduce PRP as a way to reduce the burden on LLMs. They achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs. On TREC-DL2020, PRP outperforms previous approaches based on larger models by over 5% at NDCG@1. On TREC-DL2019, PRP outperforms other existing solutions by over 10% for nearly all ranking metrics. The authors also propose several variants of PRP to improve efficiency and show that competitive results can be achieved even with linear complexity. Additionally, they highlight other benefits of PRP such as supporting both generation and scoring LLM APIs and being insensitive to input ordering. Overall, this paper presents a novel technique, PRP which significantly improves text ranking performance using moderate-sized LLMs. The results demonstrate its effectiveness in outperforming existing methods and achieving state-of-the-art performance on benchmark datasets.