# Affordances from Human Videos as a Versatile Representation for Robotics

**Authors:** Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, Deepak Pathak  
**arXiv:** [2304.08488](https://arxiv.org/abs/2304.08488)  
**Date:** 2023-04-17  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2304.08488v1/

## Key points

- Authors address the challenge of building a robot that can understand and learn to interact by observing human behavior
- Proposal to leverage videos of human interactions in an environment-centric approach
- Training a visual affordance model using internet videos of human behavior to estimate where and how humans are likely to interact within a scene
- Structured representation of behavioral affordances enables robots to perform complex tasks effectively
- Seamless integration of affordance model with four robot learning paradigms: offline imitation learning, exploration, goal-conditioned learning, and action parameterization for reinforcement learning
- Approach referred to as VRB (Visual Robotic Behavior) showcased across four real-world environments and over 10 different tasks involving two robotic platforms operating in dynamic settings
- Results highlight versatility and adaptability of method in enabling robots to learn from human interactions and perform various tasks efficiently

## Summary

In their paper titled "Affordances from Human Videos as a Versatile Representation for Robotics," authors Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak address the challenge of building a robot that can understand and learn to interact by observing human behavior. The authors propose leveraging videos of human interactions in an environment-centric approach to bridge the gap between previous static dataset research and direct application to robots. By training a visual affordance model using internet videos of human behavior, the authors aim to estimate where and how humans are likely to interact within a scene. This structured representation of behavioral affordances enables robots to perform complex tasks effectively. The authors demonstrate the seamless integration of their affordance model with four robot learning paradigms: offline imitation learning, exploration, goal-conditioned learning, and action parameterization for reinforcement learning. Their approach, referred to as VRB (Visual Robotic Behavior), is showcased across four real-world environments and encompasses over 10 different tasks involving two robotic platforms operating in dynamic settings. The results highlight the versatility and adaptability of their method in enabling robots to learn from human interactions and perform various tasks efficiently. This research was accepted at CVPR 2023 and additional information can be found on their website at https://robo-affordances.github.io/.