# Large Language Models as Optimizers

**Authors:** Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen  
**arXiv:** [2309.03409](https://arxiv.org/abs/2309.03409)  
**Date:** 2023-09-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.03409v1/

## Key points

- Proposal of Optimization by PROmpting (OPRO) as a novel approach to leverage large language models (LLMs) as optimizers
- Highlighting the challenges posed by the absence of gradient in derivative-based algorithms
- Introduction of OPRO as a simple and effective solution
- OPRO framework involves using LLMs to generate new solutions from prompts containing previously generated solutions with their values
- Evaluation and addition of new solutions to the prompt for the next optimization step
- Demonstration of effectiveness on linear regression and traveling salesman problems
- Exploration of prompt optimization to maximize task accuracy
- Comparison between prompts optimized by OPRO and human-designed prompts, showing superior performance by OPRO (up to 8% on GSM8K and up to 50% on Big-Bench Hard tasks)
- Potential of OPRO in improving optimization tasks across different domains.

## Summary

In their paper titled "Large Language Models as Optimizers," Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen propose a novel approach called Optimization by PROmpting (OPRO) to leverage large language models (LLMs) as optimizers for various real-world applications. The authors highlight the challenges posed by the absence of gradient in derivative-based algorithms and introduce OPRO as a simple and effective solution. The OPRO framework involves using LLMs to generate new solutions from prompts that contain previously generated solutions with their values. These new solutions are then evaluated and added to the prompt for the next optimization step. The authors demonstrate the effectiveness of OPRO on linear regression and traveling salesman problems. Furthermore, they explore prompt optimization where the goal is to find instructions that maximize task accuracy. They compare prompts optimized by OPRO with human-designed prompts and show that OPRO outperforms human-designed prompts by up to 8% on GSM8K and up to 50% on Big-Bench Hard tasks when using a variety of LLMs. The paper provides valuable insights into leveraging LLMs as optimizers and showcases the potential of OPRO in improving optimization tasks across different domains.