# PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs

**Authors:** Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang-Wei Edward Lee, Kuang-Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess  
**arXiv:** [2402.07872](https://arxiv.org/abs/2402.07872)  
**Date:** 2024-02-12  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2402.07872v1/

## Key points

- Vision Language Models (VLMs) have advanced capabilities in logical reasoning and visual understanding tasks.
- A novel approach called Prompting with Iterative Visual Optimization (PIVOT) is introduced to address the challenge of VLMs generating textual outputs.
- PIVOT frames tasks as iterative visual question answering processes using annotated images with visual proposals for the VLM to refine.
- PIVOT demonstrates efficacy in real-world scenarios like robotic navigation, manipulation from images, instruction following, and spatial inference tasks.
- PIVOT enables zero-shot control of robotic systems without specific training data and showcases promising capabilities in diverse environments.
- The work highlights potentials and constraints of leveraging Internet-Scale VLMs for applications in robotic and spatial reasoning domains.

## Summary

In the realm of Vision Language Models (VLMs), there has been a significant advancement in their capabilities. They have showcased impressive performance across various tasks such as logical reasoning and visual understanding. This progress has paved the way for enhanced interaction with the physical world, particularly in domains like robotic control. However, a key challenge arises from the fact that VLMs primarily generate textual outputs. To address this challenge, a novel approach known as Prompting with Iterative Visual Optimization (PIVOT) is introduced in this paper. PIVOT frames tasks as iterative visual question answering processes wherein images are annotated with visual representations of proposals that the VLM can utilize (e.g., candidate robot actions, localizations, or trajectories). Through iterative refinement of these proposals, the VLM progressively hones in on the most suitable answer for the given task. The efficacy of PIVOT is explored across real-world scenarios encompassing robotic navigation, manipulation from images, instruction following in simulation environments, and spatial inference tasks like localization. Remarkably, PIVOT demonstrates the ability to facilitate zero-shot control of robotic systems without requiring any specific training data for robots. Furthermore, it enables successful navigation across diverse environments and showcases promising capabilities despite current performance limitations. This work sheds light on both the potentials and constraints within this emerging paradigm and offers a compelling pathway towards leveraging Internet-Scale VLMs for applications in robotic and spatial reasoning domains. For further details and demonstrations of PIVOT's functionality, interested individuals can visit the project website at pivot-prompt.github.io or explore its implementation on HuggingFace at https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.