# Guiding Pretraining in Reinforcement Learning with Large Language Models

**Authors:** Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, Jacob Andreas  
**arXiv:** [2302.06692](https://arxiv.org/abs/2302.06692)  
**Date:** 2023-02-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.06692v1/

## Key points

- The paper addresses the challenge of a lack of well-defined reward function in reinforcement learning algorithms.
- The authors propose a method called ELLM that uses background knowledge from text corpora to shape exploration.
- ELLM rewards an agent for achieving goals suggested by a language model prompted with the agent's current state description.
- Large-scale language model pretraining is utilized to guide agents towards meaningful and potentially useful behaviors without human intervention.
- ELLM is evaluated in the Crafter game environment and the Housekeep robotic simulator, showing better coverage of common-sense behaviors during pretraining and improved performance on downstream tasks.
- Incorporating language models enhances exploration and learning in complex environments without explicit reward functions or human supervision.
- Leveraging text corpora can improve the efficiency and effectiveness of reinforcement learning systems.

## Summary

The paper titled "Guiding Pretraining in Reinforcement Learning with Large Language Models" addresses the challenge faced by reinforcement learning algorithms when there is a lack of a well-defined reward function. To address this issue, the authors propose a method called ELLM (Exploring with LLMs) that leverages background knowledge from text corpora to shape exploration. ELLM rewards an agent for achieving goals suggested by a language model prompted with a description of the agent's current state. By utilizing large-scale language model pretraining, ELLM guides agents towards behaviors that are both meaningful to humans and potentially useful without requiring human intervention. The authors evaluate ELLM in two different environments: the Crafter game environment and the Housekeep robotic simulator. The results show that agents trained using ELLM exhibit better coverage of common-sense behaviors during pretraining and generally match or improve performance on various downstream tasks. This approach demonstrates how incorporating language models can enhance reinforcement learning algorithms' ability to explore and learn in complex environments without relying on explicit reward functions or human supervision. The findings highlight the potential of leveraging text corpora to guide exploration and improve the efficiency and effectiveness of reinforcement learning systems.