# LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench

**Authors:** Karthik Valmeekam, Kaya Stechly, Subbarao Kambhampati  
**arXiv:** [2409.13373](https://arxiv.org/abs/2409.13373)  
**Date:** 2024-09-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.13373v1/

## Key points

- Large language models (LLMs) and their planning abilities are a focus of the study
- PlanBench introduced in 2022 as a benchmark for evaluating LLMs' planning capabilities
- OpenAI's o1 model, a Large Reasoning Model (LRM), designed to surpass traditional autoregressive LLMs
- Evaluation shows o1 model outperforms competitors on PlanBench but does not fully saturate it
- Important considerations raised regarding accuracy, efficiency, and guarantees before deploying such systems
- Study highlights ongoing debate on LLMs' planning abilities and introduces o1 model as an innovative solution to enhance reasoning capabilities within these models
- Emphasizes the need for continued research and development to unlock full potential of large language models in planning tasks

## Summary

In their paper titled "LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench," authors Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati delve into the realm of large language models (LLMs) and their planning abilities. The ability to plan a course of action to achieve a desired state has long been considered a fundamental skill of intelligent agents in AI research. With the rise of LLMs, there has been significant interest in determining whether these models possess such planning capabilities. The authors highlight the development of PlanBench in 2022 as a crucial benchmark for evaluating the planning abilities of LLMs, particularly following the release of GPT3. Despite the emergence of numerous new private and open-source LLMs post-GPT3, progress on this benchmark has been slow. OpenAI introduces their o1 (Strawberry) model as a Large Reasoning Model (LRM), specifically designed to surpass the limitations of traditional autoregressive LLMs. The paper conducts a comprehensive evaluation to assess how well current LLMs and new LRMs perform on PlanBench. While OpenAI's o1 model demonstrates significant improvement on the benchmark compared to its competitors, it falls short of fully saturating it. This advancement raises important considerations regarding accuracy, efficiency, and guarantees that must be addressed before deploying such systems. Overall, the study sheds light on the ongoing debate surrounding LLMs' planning abilities and introduces OpenAI's innovative approach with the o1 model as a potential solution to enhance reasoning capabilities within these models. The findings underscore the need for continued research and development in this area to unlock the full potential of large language models in planning tasks.