Results of the summarizing process for the arXiv paper: 2512.05962v1

, , , , In the realm of artificial intelligence, Reinforcement Learning (RL) has emerged as the go-to technique for fine-tuning Large Language Models (LLMs) to tackle tasks that require reasoning abilities. However, recent findings have indicated a notable drawback associated with models trained using RL – a significant reduction in diversity. This phenomenon is attributed to RL's implicit optimization of the "mode-seeking" or "zero-forcing" Reverse Kullback-Leibler (KL) divergence towards a target distribution. As a result, the model tends to concentrate its focus on specific high-probability regions of the target distribution while neglecting others. To address this issue, a novel approach is proposed in this study. Instead of relying on RL and its inherent biases, the researchers advocate for starting from an explicit target distribution. This distribution is derived by filtering out incorrect answers while preserving the relative probabilities of correct ones. Leveraging a pre-trained LLM as the foundation, the team approximates this refined target distribution using the $α$-divergence family. This family of divergences offers a unified framework that allows for direct control over the trade-off between precision and diversity by smoothly transitioning between mode-seeking and mass-covering divergences. The efficacy of this new methodology is put to the test on a Lean theorem-proving benchmark. The results speak volumes about its success, showcasing state-of-the-art performance along the coverage-precision Pareto frontier. Notably, this approach surpasses all previous methods when evaluated on the coverage axis, demonstrating superior capabilities in balancing both coverage and precision in reasoning tasks. Authored by Germán Kruszewski, Pierre Erbacher, Jos Rozen, and Marc Dymetman, this research paper titled "Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity" sheds light on an innovative strategy to enhance diversity and performance in LLMs through targeted filtering techniques and divergence-based optimization strategies.

• - Reinforcement Learning (RL) used for fine-tuning Large Language Models (LLMs)
• - Drawback of RL models: reduction in diversity due to mode-seeking KL divergence
• - Proposed solution: starting from an explicit target distribution derived by filtering incorrect answers while preserving relative probabilities of correct ones
• - Approach involves approximating refined target distribution using $α$-divergence family for control over precision and diversity
• - New methodology tested on Lean theorem-proving benchmark, showcasing state-of-the-art performance along coverage-precision Pareto frontier

SummaryReinforcement Learning (RL) helps make Big Language Models (LLMs) better. But sometimes RL models become less diverse because they focus too much on finding the best answer. To fix this, a new idea suggests starting with a list of correct answers and filtering out wrong ones while keeping the relative chances of each right answer. This method uses a special family of calculations to control how accurate and varied the final list is. The new way was tried on a math problem test and did really well in balancing how many problems it solved correctly versus how precise those solutions were. Definitions- Reinforcement Learning (RL): A type of learning where a computer program gets better at something by trying different actions and getting rewards for good choices. - Large Language Models (LLMs): Computer programs that can understand and generate human-like language. - Diversity: Having many different options or variations. - KL Divergence: A mathematical measure that shows how two probability distributions are different from each other. - Precision: How exact or accurate something is. - Pareto Frontier: A concept in economics showing the balance between two factors where improving one means sacrificing some of the other.

Introduction

In recent years, Reinforcement Learning (RL) has become a popular technique for fine-tuning Large Language Models (LLMs) to tackle tasks that require reasoning abilities. However, researchers have discovered a significant drawback associated with models trained using RL – a reduction in diversity. This phenomenon is attributed to RL's implicit optimization of the "mode-seeking" or "zero-forcing" Reverse Kullback-Leibler (KL) divergence towards a target distribution.

The Problem: Lack of Diversity in LLMs Trained Using RL

The team behind this research paper identified that LLMs trained using RL tend to concentrate their focus on specific high-probability regions of the target distribution while neglecting others. This lack of diversity can lead to biased and limited outputs, hindering the model's overall performance.

The Proposed Solution

To address this issue, the researchers propose a novel approach that starts from an explicit target distribution instead of relying on RL and its inherent biases. This refined target distribution is derived by filtering out incorrect answers while preserving the relative probabilities of correct ones.

Using $α$-divergence Family for Optimization

Leveraging a pre-trained LLM as the foundation, the team approximates this refined target distribution using the $α$-divergence family. This family offers a unified framework that allows for direct control over the trade-off between precision and diversity by smoothly transitioning between mode-seeking and mass-covering divergences.

Evaluation Results

The efficacy of this new methodology was put to test on a Lean theorem-proving benchmark. The results speak volumes about its success, showcasing state-of-the-art performance along the coverage-precision Pareto frontier. Notably, this approach surpasses all previous methods when evaluated on the coverage axis, demonstrating superior capabilities in balancing both coverage and precision in reasoning tasks.

Authors and Their Contributions

This research paper, titled "Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity," was authored by Germán Kruszewski, Pierre Erbacher, Jos Rozen, and Marc Dymetman. The team's combined expertise in artificial intelligence and natural language processing has led to this innovative strategy for enhancing diversity and performance in LLMs.

Conclusion

In conclusion, this research paper highlights a new approach for improving the diversity of LLMs trained using RL. By starting from an explicit target distribution and leveraging the $α$-divergence family for optimization, the proposed methodology showcases state-of-the-art performance on a Lean theorem-proving benchmark. This study opens up new possibilities for further advancements in fine-tuning LLMs for reasoning tasks.