# Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions

**Authors:** Hui Yang, Sifu Yue, Yunzhong He  
**arXiv:** [2306.02224](https://arxiv.org/abs/2306.02224)  
**Date:** 2023-06-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.02224v1/

## Key points

- Study focuses on Auto-GPT styled agents using Large Language Models (LLMs) for decision-making tasks
- Questions persist about effectiveness and adaptability of these agents in real-world scenarios
- Lack of benchmarks and limited engagement capabilities contribute to uncertainties
- Comprehensive benchmark study comparing popular LLMs (GPT-4, GPT-3.5, Claude, Vicuna) in decision-making tasks
- Introduction of Additional Opinions algorithm for supervised learning integration into Auto-GPT framework
- Algorithm significantly enhances performance in online decision-making benchmarks like WebShop and ALFWorld
- Auto-GPT surpasses state-of-the-art supervised IL models with GPT-4, showing potential for practical applications
- Additional Opinions approach holds promise for widespread adoption across industries like recommendation systems and NLP services
- Methodology can leverage LLMs for definitive determinations and explanations on item prioritization for users
- Benchmarking tasks serve as a starting point for exploring the idea, but not exhaustive of all real-world scenarios
- Adaptation of Auto-GPT through Additional Opinions paves way for further research and development in AI applications

## Summary

In this study, we delve into the realm of Auto-GPT styled agents that utilize Large Language Models (LLMs) for decision-making tasks. While there has been a surge in interest surrounding these autonomous agents, questions persist regarding their effectiveness and adaptability in real-world scenarios. The lack of benchmarks and limited engagement capabilities further add to these uncertainties. To address these concerns, we present a comprehensive benchmark study focusing on Auto-GPT styled agents in decision-making tasks that simulate real-world situations. Our objective is to gain a deeper understanding of the adaptability of GPT-based agents. We compare the performance of popular LLMs such as GPT-4, GPT-3.5, Claude, and Vicuna in Auto-GPT styled decision-making tasks. Moreover, we introduce the Additional Opinions algorithm - a simple yet effective method that integrates supervised/imitation-based learners into the Auto-GPT framework. This approach facilitates lightweight supervised learning without the need for fine-tuning the foundational LLMs. Through meticulous baseline comparisons and ablation studies, we demonstrate that the Additional Opinions algorithm significantly enhances performance in online decision-making benchmarks like WebShop and ALFWorld. Our research challenges the initial perception of Auto-GPT as merely an experimental concept by showcasing its potential for practical applications. In fact, Auto-GPT surpasses state-of-the-art supervised IL models with GPT-4, indicating a paradigm shift towards this innovative approach. We posit that the Additional Opinions approach holds promise for widespread adoption across various industries due to the prevalence of expert models such as recommendation systems and traditional NLP services. This methodology can be applied to leverage LLMs for making definitive determinations and providing explanations on item prioritization for users. While our benchmarking tasks serve as a starting point for exploring this idea, they do not encompass all potential real-world scenarios. This marks the inception of adapting Auto-GPT to handle complex tasks through Additional Opinions, paving the way for further research and development in AI applications. By expanding the practical applications of AI models like GPT-based agents, we aim to revolutionize our understanding of intricate decision-making mechanisms and their impact on diverse domains.