# Motif: Intrinsic Motivation from Artificial Intelligence Feedback

**Authors:** Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, Mikael Henaff  
**arXiv:** [2310.00166](https://arxiv.org/abs/2310.00166)  
**Date:** 2023-09-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.00166v1/

## Key points

- Motif is a method that combines prior knowledge from a Large Language Model (LLM) with reinforcement learning for training agents in decision-making tasks in rich environments.
- Motif uses the preferences of the LLM over pairs of event captions to construct an intrinsic reward for training agents.
- Motif achieves higher game scores in the NetHack game compared to algorithms trained solely to maximize scores.
- Combining Motif's intrinsic reward with the environment reward leads to better performance than existing approaches and progress on tasks without demonstrations.
- Motif generates intuitive human-aligned behaviors that can be easily modified through prompt modifications.
- Motif scales well with the size of the LLM and the amount of information given in the prompt.
- It represents a first step towards utilizing common sense and domain knowledge of LLMs to create competent AI agents without relying on complicated textual interfaces.
- The method only requires event captions and can be applied to any environment with captioning mechanisms available.
- Future work should focus not only on increasing capabilities but also deepening analysis into behavior and alignment properties.
- The paper presents Motif as a promising approach for incorporating prior knowledge from LLMs into decision-making tasks, showcasing its effectiveness in complex environments like NetHack.

## Summary

In this paper, the authors introduce Motif, a method that combines prior knowledge from a Large Language Model (LLM) with reinforcement learning to train agents for decision-making tasks in rich environments. Motif leverages the preferences of the LLM over pairs of event captions to construct an intrinsic reward, which is then used to train agents through reinforcement learning. The authors evaluate Motif's performance on the challenging and open-ended NetHack game and find that it achieves higher game scores compared to algorithms directly trained to maximize scores. Furthermore, when combining Motif's intrinsic reward with the environment reward, their method outperforms existing approaches and makes progress on tasks without demonstrations. The authors also analyze the behaviors discovered by Motif and its alignment properties. They find that Motif mostly generates intuitive human-aligned behaviors that can be easily steered through prompt modifications. Additionally, they demonstrate that Motif scales well with the size of the LLM and the amount of information given in the prompt. The authors believe that Motif represents a first step towards harnessing the common sense and domain knowledge of LLMs to create competent artificial intelligence agents. They highlight that Motif builds a bridge between an LLM's capabilities and the environment without relying on complicated textual interfaces. The method only requires event captions and can be generalized to any environment where such captioning mechanisms are available. The authors conclude by encouraging future work on similar systems to not only focus on increasing their capabilities but also deepen analysis into behavior and alignment properties. They suggest developing conceptual, theoretical, and methodological tools to align an agent's behavior in response to rewards derived from an LLM's feedback. Overall, this paper presents Motif as a promising approach for incorporating prior knowledge from LLMs into decision-making tasks, showcasing its effectiveness in complex environments like NetHack while emphasizing its potential for future advancements in AI systems.