# Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

**Authors:** Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, Matei Zaharia  
**arXiv:** [2212.14024](https://arxiv.org/abs/2212.14024)  
**Date:** 2022-12-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.14024v1/

## Key points

- Natural Language Processing (NLP) has seen significant advancements in recent years, particularly in retrieval-augmented in-context learning.
- Existing work has been limited to simple "retrieve-then-read" pipelines, where the retrieval model (RM) retrieves passages that are inserted into the language model (LM) prompt.
- A new framework called Demonstrate-Search-Predict (DSP) has been proposed to fully realize the potential of frozen LMs and RMs.
- The DSP framework relies on passing natural language texts through sophisticated pipelines between an LM and an RM to express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions by breaking down problems into small transformations that the LM and RM can handle more reliably.
- The authors have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings.
- In early evaluations, DSP has demonstrated new state-of-the-art results in context learning with relative gains against vanilla LMs ranging from 37% to 200%, a standard retrieve–then–read pipeline from 8% to 40%, and a contemporaneous self–ask pipeline from 80% to 290%.
- The authors behind this research include Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts and Matei Zaharia.
- Overall, DSP represents a powerful advancement in NLP by allowing for more sophisticated interactions between LMs and RMs.

## Summary

The field of Natural Language Processing (NLP) has seen significant advancements in recent years, particularly in the area of retrieval-augmented in-context learning. This approach involves using frozen language models (LM) and retrieval models (RM) to address knowledge-intensive tasks. However, existing work has been limited to simple "retrieve-then-read" pipelines, where the RM retrieves passages that are inserted into the LM prompt. To fully realize the potential of frozen LMs and RMs, a new framework called Demonstrate-Search-Predict (DSP) has been proposed. The DSP framework relies on passing natural language texts through sophisticated pipelines between an LM and an RM. It can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions by breaking down problems into small transformations that the LM and RM can handle more reliably. The authors have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings. In early evaluations, DSP has demonstrated new state-of-the-art results in context learning with relative gains against vanilla LMs ranging from 37% to 200%, a standard retrieve–then–read pipeline from 8% to 40%, and a contemporaneous self–ask pipeline from 80% to 290%. The authors behind this research include Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts and Matei Zaharia. Overall, DSP represents a powerful advancement in NLP by allowing for more sophisticated interactions between LMs and RMs. By breaking down complex problems into smaller steps that can be handled more effectively by these models working together within a pipeline structure designed to support them both equally well throughout each step of their interaction process - this approach is poised to revolutionize how we think about solving knowledge intensive tasks using NLP techniques.