# Are Large Language Models Good Prompt Optimizers?

**Authors:** Ruotian Ma, Xiaolei Wang, Xin Zhou, Jian Li, Nan Du, Tao Gui, Qi Zhang, Xuanjing Huang  
**arXiv:** [2402.02101](https://arxiv.org/abs/2402.02101)  
**Date:** 2024-02-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.02101v1/

## Key points

- Large Language Models (LLMs) used as Prompt Optimizers for self-reflection and prompt refinement
- LLM optimizers struggle to accurately identify root causes of errors and are biased by existing knowledge base
- Difficulty in generating appropriate prompts for target models with just one refinement step
- Focus on directly optimizing behavior of target models for more controllable results
- Proposed alternative framework shifts focus towards refining target model behavior rather than relying solely on LLMs' reflective capabilities

## Summary

By Ruotian Ma, Xiaolei Wang, Xin Zhou, Jian Li, Nan Du, Tao Gui, Qi Zhang, and Xuanjing Huang, the authors delve into the realm of LLM-based Automatic Prompt Optimization. This approach leverages Large Language Models (LLMs) as Prompt Optimizers to self-reflect and refine prompts. While showcasing promising performance in recent research endeavors, the underlying mechanism of this methodology remains largely unexplored. To address these gaps in understanding, the researchers conducted a comprehensive study aimed at uncovering the actual mechanism behind LLM-based Prompt Optimization. Their findings shed light on a critical issue: LLM optimizers often struggle to accurately identify the root causes of errors during reflection. Instead of genuinely reflecting on errors, they tend to be biased by their existing knowledge base. Moreover, even when reflections are semantically valid, LLM optimizers frequently falter in generating appropriate prompts for target models with just a single prompt refinement step. This challenge is exacerbated by the unpredictable behaviors exhibited by these target models. This innovative approach focuses on directly optimizing the behavior of target models in a more controllable manner. By shifting the focus towards refining target model behavior rather than relying solely on prompt optimization through LLMs' reflective capabilities,this new paradigm offers potential avenues for enhancing automatic prompt optimization development. Overall,this study not only highlights key limitations within current LLM-based Prompt Optimization practices but also proposes an alternative framework that could pave the way for future advancements in this field.Through their rigorous investigation and insightful conclusions,the authors aim to inspire new directions and strategies for improving automatic prompt optimization techniques moving forward.