TD-MPC2: Scalable, Robust World Models for Continuous Control

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   Mastering Atari with Discrete World Models

   Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi et al. · 2020 · Machine Learning · AI summary available

   Abstract

   Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitut…

2. Close Not relevant

   Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox

   Qiyue Yin, Tongtong Yu, Shengqi Shen et al. · 2022 · Machine Intelligence Research, 2024 (https://link.springer.com/article/10.1007/s11633-023-1454-4) · Not summarized yet

   Abstract

   With the breakthrough of AlphaGo, deep reinforcement learning becomes a recognized technique for solving sequential decision-making problems. Despite its reputation, data inefficiency caused by its trial and error learning mechanism makes deep reinforcement learning hard to be practical in a wide range of areas. Plenty of methods have been developed for sample efficient deep reinforcement learning, such as environment modeling, experience transfer, and distributed modifications, amongst which, distributed deep reinforcement learning has shown its potential in various applications, such as human-computer gaming, and intelligent transportation. In this paper, we conclude the state of this exci…

3. Close Not relevant

   Closing the Train-Test Gap in World Models for Gradient-Based Planning

   Arjun Parthasarathy, Nimit Kalra, Rohun Agrawal et al. · 2025 · Machine Learning · AI summary available

   Abstract

   World models paired with model predictive control (MPC) can be trained offline on large-scale datasets of expert trajectories and enable generalization to a wide range of planning tasks at inference time. Compared to traditional MPC procedures, which rely on slow search algorithms or on iteratively solving optimization problems exactly, gradient-based planning offers a computationally efficient alternative. However, the performance of gradient-based planning has thus far lagged behind that of other approaches. In this paper, we propose improved methods for training world models that enable efficient gradient-based planning. We begin with the observation that although a world model is trained…

4. Close Not relevant

   Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning

   Alex Beeson, Giovanni Montana · 2022 · Machine Learning · Not summarized yet

   Abstract

   The ability to discover optimal behaviour from fixed data sets has the potential to transfer the successes of reinforcement learning (RL) to domains where data collection is acutely problematic. In this offline setting, a key challenge is overcoming overestimation bias for actions not present in data which, without the ability to correct for via interaction with the environment, can propagate and compound during training, leading to highly sub-optimal policies. One simple method to reduce this bias is to introduce a policy constraint via behavioural cloning (BC), which encourages agents to pick actions closer to the source data. By finding the right balance between RL and BC such approaches …

5. Close Not relevant

   Transformers are Sample Efficient World Models

   Vincent Micheli, Eloi Alonso, François Fleuret · 2022 · Machine Learning · AI summary available

   Abstract

   Deep reinforcement learning agents are notoriously sample inefficient, which considerably limits their application to real-world problems. Recently, many model-based methods have been designed to address this issue, with learning in the imagination of a world model being one of the most prominent approaches. However, while virtually unlimited interaction with a simulated environment sounds appealing, the world model has to be accurate over extended periods of time. Motivated by the success of Transformers in sequence modeling tasks, we introduce IRIS, a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer. With the equivalent …

6. Close Not relevant

   Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training

   Mingjie Liu, Shizhe Diao, Jian Hu et al. · 2025 · Machine Learning · AI summary available

   Abstract

   Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, includ…

7. Close Not relevant

   Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes

   Aviral Kumar, Rishabh Agarwal, Xinyang Geng et al. · 2022 · Machine Learning · AI summary available

   Abstract

   The potential of offline reinforcement learning (RL) is that high-capacity models trained on large, heterogeneous datasets can lead to agents that generalize broadly, analogously to similar advances in vision and NLP. However, recent works argue that offline RL methods encounter unique challenges to scaling up model capacity. Drawing on the learnings from these works, we re-examine previous design choices and find that with appropriate choices: ResNets, cross-entropy based distributional backups, and feature normalization, offline Q-learning algorithms exhibit strong performance that scales with model capacity. Using multi-task Atari as a testbed for scaling and generalization, we train a si…

8. Close Not relevant

   Continuous control with deep reinforcement learning

   Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel et al. · 2015 · Machine Learning · Not summarized yet

   Abstract

   We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate tha…