# Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs

**Authors:** Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, Omar Khattab  
**arXiv:** [2406.11695](https://arxiv.org/abs/2406.11695)  
**Date:** 2024-06-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.11695v1/

## Key points

- Authors delve into Language Model Programs (LM), which are revolutionizing NLP tasks
- Study focuses on prompt optimization for LM programs without module-level labels or gradients
- Problem is broken down into optimizing free-form instructions and few-shot demonstrations for each module
- Strategies introduced include crafting task-specific instructions, assigning credit across modules, and utilizing stochastic mini-batch evaluation function
- Meta-optimization procedure implemented to enhance how LMs generate proposals over time
- MIPRO, a novel optimizer developed by authors, showcases superior performance compared to baseline methods
- Achieves up to 12.9% higher accuracy on five out of six LM programs evaluated
- Authors plan to release new optimizers and benchmark in DSPy at https://github.com/stanfordnlp/dspy

## Summary

In their paper titled "Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs," authors Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab delve into the realm of Language Model Programs (LM) - intricate pipelines of modular LM calls that are revolutionizing Natural Language Processing (NLP) tasks. The focus of the study is prompt optimization for LM programs, specifically exploring how prompts can be updated to enhance downstream metrics without access to module-level labels or gradients. To tackle this challenge, the authors break down the problem into optimizing both the free-form instructions and few-shot demonstrations associated with each module. They introduce a range of strategies aimed at crafting task-specific instructions and efficiently assigning credit across modules. These strategies encompass program- and data-aware techniques for proposing effective instructions, utilizing a stochastic mini-batch evaluation function to learn a surrogate model of the objective, and implementing a meta-optimization procedure to enhance how LMs generate proposals over time. The culmination of these insights is MIPRO, a novel optimizer developed by the authors. Through rigorous experimentation on six diverse LM programs using the state-of-the-art open-source model Llama-3-8B, MIPRO showcases superior performance compared to baseline methods. In fact,MIPRO achieves up to 12.9% higher accuracy on five out of six LM programs evaluated. The authors also announce their intention to release their new optimizers and benchmark in DSPy at https://github.com/stanfordnlp/dspy. This comprehensive study not only sheds light on the intricacies of prompt optimization in LM programs but also presents a practical solution in MIPRO that significantly enhances performance across various NLP tasks.