# Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

**Authors:** Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, Peter Stone  
**arXiv:** [2003.04960](https://arxiv.org/abs/2003.04960)  
**Date:** 2020-03-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2003.04960v2/

## Key points

- Reinforcement learning (RL) is used for solving sequential decision tasks with limited feedback from the environment.
- Transfer learning and curriculum learning have been applied to RL to address the challenge of extensive interaction with the environment.
- Transfer learning allows leveraging experience from one task to learn a more difficult task, accelerating learning in new tasks.
- Curriculum learning involves sequencing tasks or data samples into a curriculum to facilitate learning of complex problems.
- The authors present a framework for curriculum learning in RL and provide a comprehensive survey and classification of existing methods.
- Open problems and future research directions in RL curriculum learning are identified.
- Acknowledgments are given to individuals and organizations that supported the research.
- Two specific approaches are discussed: using an object-oriented approach for creating source tasks and using progression functions to determine task difficulty changes over time.
- Leveraging prior knowledge and sequencing tasks improve learning efficiency in RL domains.

## Summary

Reinforcement learning (RL) is a widely used approach for solving sequential decision tasks where the agent has limited feedback from the environment. However, in many domains, extensive interaction with the environment is still necessary for effective learning. This can be impractical in realistic scenarios. To address this challenge, transfer learning and curriculum learning have been applied to RL. Transfer learning allows experience gained from one task to be leveraged when learning a more difficult task. This approach enables the agent to build upon previously acquired knowledge and accelerate learning in new tasks. On the other hand, curriculum learning involves sequencing tasks or data samples into a curriculum to facilitate learning of complex problems that may be difficult to learn from scratch. In this article, the authors present a framework for curriculum learning in RL and provide a comprehensive survey and classification of existing curriculum learning methods based on their assumptions, capabilities, and goals. The authors also identify open problems and suggest future research directions in RL curriculum learning. The acknowledgments section acknowledges the contributions of Brad Knox, Garrett Warnell, anonymous reviewers, and various organizations that have supported the research. The work has been conducted at different research groups such as the Learning Agents Research Group (LARG) at The University of Texas at Austin, Sensible Robots Research Group at the University of Leeds, Control, Robotics, Identification and Signal Processing (CRISP) Laboratory at Tufts University,

Whiteson Research Lab at the University of Oxford,

and Intelligent Robot Learning (IRL) Lab at the University of Alberta. Two specific approaches are discussed in more detail within the expanded content: one approach formalizes task feature descriptors using an object-oriented approach to create source tasks for curriculum generation; another approach considers how long to spend on each intermediate task in the curriculum by using progression functions that specify the pace at which task difficulty should change over time. Overall, this expanded summary provides a deeper understanding of reinforcement learning with a focus on transfer learning and curriculum learning methods. It highlights the importance of leveraging prior knowledge and sequencing tasks to improve learning efficiency in RL domains.