# On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models

**Authors:** Mudit Verma, Siddhant Bhambri, Subbarao Kambhampati  
**arXiv:** [2405.13966](https://arxiv.org/abs/2405.13966)  
**Date:** 2024-05-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.13966v1/

## Key points

- The study examines the reasoning abilities of Large Language Models (LLMs) with a focus on ReAct-based prompting methods.
- There is uncertainty about how ReAct-based prompting enhances sequential decision-making in agentic LLMs.
- Experiments show that factors like interleaving reasoning trace with action execution or content of generated reasoning traces have minimal impact on performance.
- Perceived reasoning abilities may be influenced by exemplar-query similarity and approximate retrieval rather than inherent capabilities.
- Performance in tasks like AlfWorld is heavily influenced by how closely example tasks align with query tasks.
- Trivial variations in exemplar prompts can significantly affect performance, cautioning against uncritical adoption of ReAct-style frameworks.
- The research emphasizes the need to critically evaluate prompt-engineering methods claiming emergent abilities in LLMs across various sectors such as economics, healthcare, and transport.

## Summary

The study "On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models" delves into the debate surrounding the reasoning abilities of Large Language Models (LLMs). Specifically, it focuses on ReAct-based prompting methods and their supposed enhancement of sequential decision-making capabilities in agentic LLMs. However, there is a lack of clarity regarding the source of improvement in LLM reasoning with ReAct-based prompting. Through a series of experiments involving systematic variations to input prompts, the researchers conduct a sensitivity analysis to scrutinize these claims. Contrary to expectations, they find that performance is minimally impacted by factors such as interleaving reasoning trace with action execution or the content of generated reasoning traces in ReAct. This challenges the original claims and common usage patterns associated with ReAct. The findings suggest that perceived reasoning abilities may not stem from inherent capabilities but rather from factors such as exemplar-query similarity and approximate retrieval. The study highlights that performance in sequential decision-making tasks like AlfWorld is heavily influenced by how closely example tasks align with query tasks. Additionally, it reveals that trivial variations in exemplar prompts can significantly affect performance, cautioning against uncritical adoption of ReAct-style frameworks. In conclusion, this research underscores the importance of critically examining prompt-engineering methods claiming emergent abilities in LLMs. The implications extend to various sectors such as economics, healthcare, and transport. While this study focuses on a specific domain and problem type, it serves as a call for similar scrutiny across different domains utilizing prompting solutions for reasoning tasks. Ultimately, this work aims to improve experimentation standards within the agentic LLM community and promote a more nuanced understanding of prompt engineering techniques.