# JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models

**Authors:** Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, Yitao Liang  
**arXiv:** [2311.05997](https://arxiv.org/abs/2311.05997)  
**Date:** 2023-11-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.05997v2/

## Key points

- JARVIS-1 is an open-world agent developed for the Minecraft universe
- It can perceive multimodal input, generate plans, and perform control tasks
- Utilizes pre-trained language models to map visual observations and textual instructions to plans
- Equipped with a multimodal memory combining pre-trained knowledge and game survival experiences
- Capable of completing over 200 different tasks in Minecraft, ranging from short to long horizon tasks
- Outperforms existing state-of-the-art agents in long term tasks by five times
- Demonstrates superior generalization and planning abilities
- Method used does not rely on fine tuning through imitation learning or reinforcement learning
- Maintains a high success rate even as the duration of the task increases
- Incorporates self-checking mechanisms to ensure plan correctness
- Can dynamically re-plan based on current inventory and game conditions

## Summary

JARVIS-1 is an open-world agent developed for the Minecraft universe that can perceive multimodal input, generate plans and perform control tasks. It utilizes pre-trained language models to map visual observations and textual instructions to plans which are then executed by goal-conditioned controllers. The agent is equipped with a multimodal memory combining pre-trained knowledge and its own game survival experiences. JARVIS-1 is capable of completing over 200 different tasks in Minecraft ranging from short to long horizon tasks. It outperforms existing state-of-the art agents in long term tasks by five times and demonstrates superior generalization and planning abilities. The method used does not rely on fine tuning through imitation learning or reinforcement learning. Furthermore, it maintains a high success rate even as the duration of the task increases unlike other approaches that struggle with longer time frames. Additionally, JARVIS-1 incorporates self checking mechanisms to ensure plan correctness and can dynamically re plan based on its current inventory and game conditions.