# Transformers are Sample Efficient World Models

**Authors:** Vincent Micheli, Eloi Alonso, François Fleuret  
**arXiv:** [2209.00588](https://arxiv.org/abs/2209.00588)  
**Date:** 2022-09-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2209.00588v1/

## Key points

- Deep reinforcement learning agents have been limited in their application to real-world problems due to sample inefficiency
- Model-based methods have been designed to address this issue, with learning in the imagination of a world model being one of the most prominent approaches
- Ensuring that the world model is accurate over extended periods of time has been a challenge
- IRIS is a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer
- IRIS achieves remarkable results on the Atari 100k benchmark with only two hours of gameplay equivalent training time, outperforming humans on 10 out of 26 games and setting a new state-of-the-art for methods without lookahead search and even surpassing MuZero
- The success of IRIS is attributed to its ability to learn from sequences efficiently using Transformers while also leveraging information from past interactions through its discrete autoencoder component
- The authors release their codebase at https://github.com/eloialonso/iris to foster future research on Transformers and world models for sample-efficient reinforcement learning.

## Summary

Deep reinforcement learning agents have been limited in their application to real-world problems due to their sample inefficiency. To address this issue, many model-based methods have been designed, with learning in the imagination of a world model being one of the most prominent approaches. However, ensuring that the world model is accurate over extended periods of time has been a challenge. In this context, the authors introduce IRIS, a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer. The authors were motivated by the success of Transformers in sequence modeling tasks and sought to apply them to sample-efficient reinforcement learning. IRIS achieves remarkable results on the Atari 100k benchmark with only two hours of gameplay equivalent training time. It achieves a mean human normalized score of 1.046 and outperforms humans on 10 out of 26 games, setting a new state-of-the-art for methods without lookahead search and even surpassing MuZero. The success of IRIS is attributed to its ability to learn from sequences efficiently using Transformers while also leveraging information from past interactions through its discrete autoencoder component. By combining these two components, IRIS can accurately predict future states given past observations while also generating diverse imagined trajectories. To foster future research on Transformers and world models for sample-efficient reinforcement learning, the authors release their codebase at https://github.com/eloialonso/iris. Overall, this work demonstrates how transformers can be used effectively in deep reinforcement learning agents and opens up new avenues for research in this area.