# Learning to Retrieve In-Context Examples for Large Language Models

**Authors:** Liang Wang, Nan Yang, Furu Wei  
**arXiv:** [2307.07164](https://arxiv.org/abs/2307.07164)  
**Date:** 2023-07-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.07164v1/

## Key points

- Large Language Models (LLMs) have impressive in-context learning abilities
- In-context learning relies on the quality of selected examples
- Proposed framework trains dense retrievers to identify high-quality in-context examples for LLMs
- Uses knowledge distillation and a reward model based on LLM feedback
- Experimental results show significant improvements in in-context learning performance across 30 tasks
- Model improves performance by retrieving examples with similar patterns
- Quality of evaluation LLM has a greater impact on final performance than ranking LLM
- Increasing the number of examples and retriever size leads to improved performance
- Outperforms several baselines across various tasks such as close QA, commonsense reasoning, coreference resolution, NLI, paraphrasing, reading comprehension, sentiment analysis, data-to-text generation, and summarization

## Summary

, , , , 


Large Language Models, In-Context Learning, Dense Retrievers, Knowledge Distillation, Generalization Ability

In recent years, Large Language Models (LLMs) have demonstrated their impressive ability to learn in-context and perform various tasks with only a few input-output examples. However, the effectiveness of this in-context learning heavily relies on the quality of selected examples. To address this issue, the authors propose a novel framework for training dense retrievers that can identify high-quality in-context examples for LLMs. This involves training a reward model based on LLM feedback to evaluate candidate examples and using knowledge distillation to train a bi-encoder based dense retriever. Experimental results on 30 tasks show that the proposed framework significantly enhances in-context learning performance and exhibits generalization ability to unseen tasks during training. Further analysis reveals that the model improves performance by retrieving examples with similar patterns, and these gains are consistent across LLMs of different sizes. The authors also investigate the impacts of using different LLMs for candidate ranking and task evaluation, finding that the quality of the evaluation LLM has a greater impact on final performance compared to the choice of ranking LLM. Additionally, they explore scaling effects with respect to the number of in-context examples and retriever size, finding that increasing both leads to improved performance. Overall, these results demonstrate that the proposed framework outperforms several baselines across various tasks such as close QA, commonsense reasoning, coreference resolution, NLI, paraphrasing, reading comprehension, sentiment analysis, data-to-text generation and summarization. In conclusion,<kgd> this paper presents a novel framework for training dense retrievers to identify high-quality in-context examples for LLMs. The experimental results demonstrate significant improvements in in-context learning performance across various tasks, showcasing the potential of this approach. Furthermore, the proposed framework exhibits generalization ability and consistent gains across different LLM sizes, making it a promising direction for future research.