# A NotSo Simple Way to Beat Simple Bench

**Authors:** Soham Sane, Angus McLean  
**arXiv:** [2412.12173](https://arxiv.org/abs/2412.12173)  
**Date:** 2024-12-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.12173v1/

## Key points

- A novel framework enhances reasoning capabilities of large language models (LLMs) through iterative reasoning and feedback-driven methodologies.
- Proposed approach involves multi-step prompting strategy and global consistency checks to improve model accuracy and robustness.
- Comparative analysis shows that iterative reasoning significantly boosts model performance, with improvements in standard accuracy metrics (AVG@5) and Extreme Averaging (EAG@5).
- Specific strengths of leading models highlighted: Claude excels in logical consistency, GPT-4o showcases exploratory creativity but struggles with ambiguous prompts.
- Research underscores areas for further refinement in spatial and temporal reasoning, emphasizing potential of structured reasoning frameworks to overcome model limitations.
- Study lays groundwork for integrating dynamic feedback mechanisms, adaptive restart strategies, and diverse evaluation metrics to advance LLM reasoning capabilities across complex problem spaces.

## Summary

This paper presents a novel framework that enhances the reasoning capabilities of large language models (LLMs) by incorporating iterative reasoning and feedback-driven methodologies. The research addresses limitations identified in the SimpleBench benchmark for evaluating logical coherence and real-world reasoning. The proposed approach involves a multi-step prompting strategy combined with global consistency checks to improve model accuracy and robustness. A comparative analysis of leading models such as Claude 3 Opus, Claude 3.5, GPT-4o, and o1-preview demonstrates that iterative reasoning significantly boosts model performance. Improvements are observed in standard accuracy metrics (AVG@5) as well as a newly introduced metric called Extreme Averaging (EAG@5). The results highlight specific strengths of each model; for instance, Claude excels in maintaining logical consistency while GPT-4o showcases exploratory creativity but struggles with ambiguous prompts. Through case studies and identification of gaps in spatial and temporal reasoning, the research underscores areas for further refinement. It emphasizes the potential of structured reasoning frameworks to address inherent model limitations regardless of pretraining methodologies. The study lays the groundwork for integrating dynamic feedback mechanisms, adaptive restart strategies, and diverse evaluation metrics to advance LLM reasoning capabilities across complex problem spaces. The background section provides context on the challenges faced by LLMs in maintaining logical coherence and navigating multi-step problem-solving scenarios. Existing methods like direct inference or chain-of-thought prompting have shown partial success but often struggle with global consistency and adapting to ambiguous problems. The paper addresses these challenges by proposing an iterative reasoning framework that incorporates multi-step prompting, feedback validation, and global consistency checks. This structured process aims to enhance logical coherence, adaptability, and overall robustness of LLMs when tackling complex reasoning tasks. Related work is discussed, highlighting previous efforts such as Chain-of-Thought (CoT) prompting and iterative CoT prompting with feedback loops. While these approaches have shown promise in improving LLM reasoning capabilities, they have limitations in scalability and handling complex tasks consistently. The primary goals of this research are to overcome existing limitations in reasoning methodologies by introducing an iterative framework that enhances model performance through dynamic adaptability and refined reasoning processes.