# Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision

**Authors:** Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, Xiao Wang, Rui Zheng, Tao Ji, Xiaowei Shi, Yitao Zhai, Rongxiang Weng, Jingang Wang, Xunliang Cai, Tao Gui, Zuxuan Wu  
**arXiv:** [2411.16579](https://arxiv.org/abs/2411.16579)  
**Date:** 2024-11-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.16579v1/

## Key points

- Training large language models (LLMs) for complex reasoning tasks in domains like science, coding, and mathematics requires thoughtful reflection before responding.
- Success of self-reflection and self-correction mechanisms depends on the model's ability to evaluate its own performance accurately.
- Factors hindering accurate evaluation include initial accuracy, question complexity, and lack of external feedback.
- A two-player framework is explored where critique models provide step-level feedback to supervise reasoning models during test-time and train-time.
- AutoMathCritique is introduced as an automated framework for collecting critique data, resulting in a dataset with detailed step-level feedback paired with responses.
- Fine-tuning language models using this dataset enables natural language feedback generation for mathematical reasoning tasks.
- Critique models consistently enhance actor's performance on challenging queries at test-time, especially with increased inference-time computation.
- Incorporating critique-based supervision into actor's self-training process leads to enhanced exploration efficiency and solution diversity on difficult queries - resulting in a more robust reasoning model.
- The study explores training self-talk reasoning models via critique supervision and highlights potential benefits.

## Summary

Training large language models (LLMs) to engage in more thoughtful reflection before responding is essential for effectively tackling complex reasoning tasks in domains such as science, coding, and mathematics. The success of mechanisms like self-reflection and self-correction hinges on the model's ability to accurately evaluate its own performance. However, this can be hindered by factors like initial accuracy, question complexity, and a lack of external feedback. In this study, a two-player framework is explored where reasoning and critique models are separated roles. The critique model provides step-level feedback to supervise the reasoning (actor) model during both test-time and train-time. The researchers introduce AutoMathCritique - an automated and scalable framework designed for collecting critique data. This results in a dataset comprising 76,321 responses paired with detailed step-level feedback. By fine-tuning language models using this dataset, they enable these models to generate natural language feedback for mathematical reasoning tasks. Results demonstrate that the critique models consistently enhance the actor's performance on challenging queries at test-time, particularly when increasing inference-time computation. Building upon these findings, the researchers incorporate critique-based supervision into the actor's self-training process and propose a critique-in-the-loop self-improvement method. Experimental outcomes indicate that this method enhances the actor's exploration efficiency and solution diversity, especially on difficult queries - leading to a more robust reasoning model. Furthermore, the study takes an initial step towards exploring training self-talk reasoning models via critique supervision and showcases its potential benefits. The code and datasets associated with this research are available at https://mathcritique.github.io/. The team behind this work includes Zhiheng Xi, Dingwen Yang, Jixuan Huang,Jiafu Tang,GuanYu Li,Yiwen Ding Wei He Boyang Hong Shihan Do Wenyu Zhan Xiao Wang Rui Zheng Tao Ji Xiaowei Shi Yitao Zhai Rongxiang Weng Jingang Wang Xunliang Cai Tao Gui Zuxuan Wu Qi Zhang among others.