Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Pas pertinent

   Guiding Pretraining in Reinforcement Learning with Large Language Models

   Yuqing Du, Olivia Watkins, Zihan Wang et al. · 2023 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Reinforcement learning algorithms typically struggle in the absence of a dense, well-shaped reward function. Intrinsically motivated exploration methods address this limitation by rewarding agents for visiting novel states or transitions, but these methods offer limited benefits in large environments where most discovered novelty is irrelevant for downstream tasks. We describe a method that uses background knowledge from text corpora to shape exploration. This method, called ELLM (Exploring with LLMs) rewards an agent for achieving goals suggested by a language model prompted with a description of the agent's current state. By leveraging large-scale language model pretraining, ELLM guides ag…

2. Très proche Pas pertinent

   Teaching Large Language Models to Reason with Reinforcement Learning

   Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy et al. · 2024 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Reinforcement Learning from Human Feedback (\textbf{RLHF}) has emerged as a dominant approach for aligning LLM outputs with human preferences. Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback (Expert Iteration, Proximal Policy Optimization (\textbf{PPO}), Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate both sparse and dense rewards provided to the LLM both heuristically and via a learned reward model. We additionally start from multiple model sizes and initializations both with and without supervised fine-tuning (\textbf{SFT}) data. Overall, we find all algorithms perform comparably, with Expert Iteration…

3. Très proche Pas pertinent

   Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

   Joey Hong, Sergey Levine, Anca Dragan · 2023 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. For example, a teacher might try to understand their student's current comprehension level to tailor their instruction accordingly, and a travel agent might ask questions of their customer to understand their preferences in order to recommend activities they might enjoy. LLMs trained with supervised fine-tuning or "single-step" RL, as with standard RLHF, might struggle which tasks that require such goal-directed behavior, since they…

4. Proche Pas pertinent

   Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training

   Mingjie Liu, Shizhe Diao, Jian Hu et al. · 2025 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, includ…

5. Proche Pas pertinent

   AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs

   Huichi Zhou, Yihang Chen, Siyuan Guo et al. · 2025 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   In this paper, we introduce a novel learning paradigm for adaptive Large Language Model (LLM) agents that eliminates the need for fine-tuning the underlying LLMs. Existing approaches are often either rigid, relying on static, handcrafted reflection workflows, or computationally intensive, requiring gradient updates of LLM model parameters. In contrast, our method enables low-cost continual adaptation via memory-based online reinforcement learning. We formalise this as a Memory-augmented Markov Decision Process (M-MDP), equipped with a neural case-selection policy to guide action decisions. Past experiences are stored in an episodic memory, either differentiable or non-parametric. The policy …

6. Proche Pas pertinent

   Foundational Challenges in Assuring Alignment and Safety of Large Language Models

   Usman Anwar, Abulhair Saparov, Javier Rando et al. · 2024 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are organized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose $200+$ concrete research questions.

7. Proche Pas pertinent

   Stabilizing Reinforcement Learning with LLMs: Formulation and Practices

   Chujie Zheng, Kai Dang, Bowen Yu et al. · 2025 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Exp…

8. Proche Pas pertinent

   Large Language Model Can Continue Evolving From Mistakes

   Haokun Zhao, Haixia Han, Jie Shi et al. · 2024 · Apprentissage automatique · Résumé IA disponible

   Résumé scientifique

   Large Language Models (LLMs) demonstrate impressive performance in various downstream tasks. However, they may still generate incorrect responses in certain scenarios due to the knowledge deficiencies and the flawed pre-training data. Continual Learning (CL) is a commonly used method to address this issue. Traditional CL is task-oriented, using novel or factually accurate data to retrain LLMs from scratch. However, this method requires more task-related training data and incurs expensive training costs. To address this challenge, we propose the Continue Evolving from Mistakes (CEM) method, inspired by the 'summarize mistakes' learning skill, to achieve iterative refinement of LLMs. Specifica…