# Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents

**Authors:** Yue Wu, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, Shrimai Prabhumoye  
**arXiv:** [2305.02412](https://arxiv.org/abs/2305.02412)  
**Date:** 2023-05-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.02412v2/

## Key points

- The study focuses on leveraging pre-trained large language models (LLMs) to simplify complex control tasks without compromising the trainable nature of the actor.
- The proposed Plan, Eliminate, and Track (PET) framework consists of three key modules: Plan, Eliminate, and Track.
- The Plan module breaks down tasks into sub-tasks using a pre-trained LLM.
- The Eliminate module masks out irrelevant objects and receptacles from observations for the current sub-task using a zero-shot QA language model.
- The Track module determines task completion and transitions to the next sub-task.
- An Action Attention agent based on a transformer architecture is introduced to handle changing action spaces in text environments.
- Results show that LLMs can remove 40% of task-irrelevant objects through common-sense QA and generate high-level sub-tasks with 99% accuracy.
- Coordination between multiple LLMs can assist agents from different perspectives.
- Contributions include introducing the PET framework as a novel approach to leveraging pre-trained LLMs with embodied agents.
- Each component of P, E, T plays a complementary role in addressing control tasks effectively.
- An Action Attention agent is introduced to handle variable length action spaces in text environments.
- There is a significant 15% improvement over state-of-the-art methods for generalization to human goals through sub-task planning and tracking.

## Summary

This study focuses on leveraging pre-trained large language models (LLMs) to simplify complex control tasks without compromising the trainable nature of the actor. The proposed Plan, Eliminate, and Track (PET) framework consists of three key modules: Plan, Eliminate, and Track. The Plan module breaks down tasks into sub-tasks using a pre-trained LLM. The Eliminate module masks out irrelevant objects and receptacles from observations for the current sub-task using a zero-shot QA language model. Lastly, the Track module determines task completion and transitions to the next sub-task. Additionally, an Action Attention agent based on a transformer architecture is introduced to handle changing action spaces in text environments. This study specifically explores instruction following in indoor households within the AlfWorld interactive text environment benchmark. Results show that LLMs can remove 40% of task-irrelevant objects through common-sense QA and generate high-level sub-tasks with 99% accuracy. Furthermore, coordination between multiple LLMs can assist agents from different perspectives. The contributions of this work include introducing the PET framework as a novel approach to leveraging pre-trained LLMs with embodied agents. The study demonstrates that each component of P, E, T plays a complementary role in addressing control tasks effectively. Additionally, an Action Attention agent is introduced to handle variable length action spaces in text environments. Overall, there is a significant 15% improvement over state-of-the-art methods for generalization to human goals through sub-task planning and tracking. In related work analysis, prior research on language-conditioned policies through imitation learning or reinforcement learning has been explored. While some studies have used pre-trained language embeddings to enhance generalization to new instructions, they lack domain knowledge captured in LLMs. The PET framework enables effective planning, progress tracking, and observation filtering by harnessing the capabilities of LLMs in simplifying complex control tasks without compromising the trainable nature of the actor.