# LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning

**Authors:** Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, Wanli Ouyang, Dongzhan Zhou  
**arXiv:** [2410.02884](https://arxiv.org/abs/2410.02884)  
**Date:** 2024-10-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.02884v1/

## Key points

- LLaMA-Berry is a mathematical problem-solving framework designed to enhance the reasoning abilities of Large Language Models (LLMs) like GPT-4 for tackling complex Olympiad-level benchmarks such as AIME.
- The framework combines Monte Carlo Tree Search (MCTS) with iterative Self-Refine to optimize the reasoning path and uses a pairwise reward model to globally evaluate different paths.
- Self-Refine applied to MCTS (SR-MCTS) overcomes limitations of conventional search algorithms by facilitating more efficient exploration of solution spaces using LLMs' self-critic and rewriting capabilities.
- The Pairwise Preference Reward Model (PPRM) is employed to model preferences between solutions, addressing scoring variability and non-independent distributions in mathematical reasoning tasks.
- LLaMA-Berry demonstrates superior performance in search efficiency and problem-solving capability compared to existing methods on challenging Olympiad-level benchmarks like GPQA, AIME24, and AMC23.

## Summary

, , , , 

This paper presents LLaMA-Berry, an innovative mathematical problem-solving framework designed to enhance the reasoning abilities of Large Language Models (LLMs) such as GPT-4 in tackling complex Olympiad-level benchmarks like AIME. The framework combines Monte Carlo Tree Search (MCTS) with iterative Self-Refine to optimize the reasoning path and utilizes a pairwise reward model to globally evaluate different paths. By leveraging the self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS (SR-MCTS) overcomes limitations of conventional step-wise and greedy search algorithms by facilitating more efficient exploration of solution spaces. Inspired by Reinforcement Learning from Human Feedback (RLHF), the Pairwise Preference Reward Model~(PPRM) is employed to model pairwise preferences between solutions, using an Enhanced Borda Count (EBC) method to synthesize these preferences into a global ranking score for finding better answers. This approach effectively addresses scoring variability and non-independent distributions in mathematical reasoning tasks. The framework has been rigorously tested on general and advanced benchmarks, demonstrating superior performance in terms of search efficiency and problem-solving capability compared to existing methods like ToT and rStar. Particularly in challenging Olympiad-level benchmarks including GPQA, AIME24, and AMC23, LLaMA-Berry showcases its prowess in generating effective reasoning paths for solving complex mathematical problems. Furthermore, a scaling study was conducted to explore the impact of test-time rollouts on model performance across different difficulty levels. The results indicate that increasing the number of rollouts consistently enhances model performance, with variations based on benchmark complexity and the base model's reasoning capability. The study also highlights how different models within the framework respond to increased computational support, emphasizing the importance of foundational model capabilities in handling intricate mathematical reasoning tasks. In conclusion, LLaMA-Berry presents a cutting-edge approach to enhancing mathematical reasoning in LLMs through optimized search algorithms and pairwise preference modeling. Its success in addressing challenges posed by Olympiad-level benchmarks underscores its potential for advancing artificial intelligence capabilities in mathematical problem-solving domains.