# DiffGen: Robot Demonstration Generation via Differentiable Physics Simulation, Differentiable Rendering, and Vision-Language Model

**Authors:** Yang Jin, Jun Lv, Shuqiang Jiang, Cewu Lu  
**arXiv:** [2405.07309](https://arxiv.org/abs/2405.07309)  
**Date:** 2024-05-12  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2405.07309v1/

## Key points

- Generating robot demonstrations through simulation is an effective method to scale up robot data in the field of robotics.
- Previous approaches using reinforcement learning agents lacked sample efficiency.
- DiffGen integrates differentiable physics simulation, differentiable rendering, and a vision-language model for automatic and efficient generation of robot demonstrations.
- The key innovation of DiffGen lies in minimizing the distance between natural language instructions and simulated observations to generate realistic robot demonstrations.
- DiffGen enables efficient and effective generation of robot data with minimal human effort or training time, reducing reliance on manual reward design processes.

## Summary

, , , , 

In the field of robotics, generating robot demonstrations through simulation is widely acknowledged as an effective method to scale up robot data. Previous approaches often involved training reinforcement learning agents to generate expert policies, but were found to lack sample efficiency. However, a new line of research has emerged that focuses on generating robot demonstrations through differentiable simulation. This approach heavily relies on reward design, which can be a time-consuming and labor-intensive process. DiffGen integrates differentiable physics simulation, differentiable rendering, and a vision-language model to facilitate automatic and efficient generation of robot demonstrations. The key innovation lies in its ability to generate realistic robot demonstrations by minimizing the distance between the embedding of a natural language instruction and the embedding of the simulated observation after manipulation. These embeddings are derived from the vision-language model used in the framework. The optimization process involves calculating and descending gradients through the differentiable simulation, differentiable rendering, and vision-language model components to achieve the specified task. Experimental results demonstrate that with DiffGen, it is possible to efficiently and effectively generate robot data with minimal human effort or training time. This framework offers a promising solution for automating the generation of robot demonstrations while reducing reliance on manual reward design processes. Overall, DiffGen represents a significant advancement in enabling automated generation of realistic robot demonstrations through innovative integration of various components within a unified framework.