# Robustness Assessment of Mathematical Reasoning in the Presence of Missing and Contradictory Conditions

**Authors:** Shi-Yu Tian, Zhi Zhou, Lin-Han Jia, Lan-Zhe Guo, Yu-Feng Li  
**arXiv:** [2406.05055](https://arxiv.org/abs/2406.05055)  
**Date:** 2024-06-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.05055v1/

## Key points

- Few-shot prompting techniques have enhanced the performance of Large Language Models (LLMs) on reasoning tasks.
- Existing evaluations focus on well-structured benchmarks and neglect real-world reasoning problems with missing and contradictory conditions, known as ill-defined problems.
- The gap in evaluation has shown that current few-shot prompting methods struggle with handling ill-defined problems, leading to overconfident answers or hallucinations.
- A new benchmark called Problems with Missing and Contradictory Conditions (PMC) has been developed to assess few-shot prompting methods' performance on ill-defined problems, introducing two novel metrics for evaluation.
- A trade-off dilemma exists between mathematical reasoning for well-defined problems and recognizing ill-defined problems when using few-shot prompting methods.
- The SMT-LIB Prompting (SLP) method has been proposed to address these challenges by utilizing the SMT-LIB language to model problems instead of solving them directly. It employs a double-check solving strategy for verifying solutions' satisfiability and uniqueness, leading to more accurate feedback.
- Extensive experiments have demonstrated the superiority of the SLP approach compared to existing few-shot prompting methods in tackling problems with missing and contradictory conditions.
- Related work includes advancements in CoT-type methods, program-type methods, ensemble-optimized approaches for few-shot prompting techniques; strategies like Metamath, WizardMath, Mugglemath, Mathvista for enhancing mathematical reasoning skills; and research on perturbations to model inputs and noisy ground truth prompting for LLM robustness.

## Summary

In the realm of Large Language Models (LLMs), the utilization of few-shot prompting techniques has significantly enhanced their performance on reasoning tasks. However, existing evaluations primarily focus on well-structured benchmarks and neglect real-world reasoning problems that often present missing and contradictory conditions - known as ill-defined problems. This gap in evaluation has highlighted the ineffectiveness of current few-shot prompting methods in handling such scenarios, leading to overconfident answers or hallucinations. To address this issue, a new benchmark called Problems with Missing and Contradictory Conditions (PMC) has been developed. This benchmark aims to assess the performance of few-shot prompting methods in dealing with ill-defined problems by introducing two novel metrics for evaluation. Through analysis using the PMC benchmark, a trade-off dilemma has been identified between mathematical reasoning for well-defined problems and the recognition of ill-defined problems. In response to these challenges, a novel few-shot prompting method called SMT-LIB Prompting (SLP) has been proposed. SLP utilizes the SMT-LIB language to model problems instead of solving them directly. A double-check solving strategy is employed to verify the satisfiability and uniqueness of solutions, providing more accurate feedback. Extensive experiments have demonstrated the superiority of the SLP approach compared to existing few-shot prompting methods when tackling problems with missing and contradictory conditions. Furthermore, our paper delves into related work in three key areas: few-shot prompting methods, mathematical reasoning for LLMs, and natural language benchmarks for LLM robustness. The study highlights advancements in CoT-type methods, program-type methods, ensemble-optimized approaches for few-shot prompting techniques; strategies like Metamath, WizardMath, Mugglemath, Mathvista for enhancing mathematical reasoning skills; and research on perturbations to model inputs and noisy ground truth prompting for LLM robustness. Overall, this comprehensive exploration sheds light on the challenges posed by ill-defined reasoning problems and introduces innovative solutions through the SLP method. The open-sourcing of our benchmark and code will facilitate further research in this evolving field.