Learning Object Manipulation Skills from Video via Approximate Differentiable Physics

Vladimir Petrik, Mohammad Nomaan Qureshi, Josef Sivic, and Makarand Tapaswi

CIIRC, Czech Technical University in Prague CVIT, IIIT Hyderabad, India

This page presents supplementary materials for our IROS 2022 submission on Learning Object Manipulation Skills from Video via Approximate Differentiable Physics. In case of any question contact us at vladimir.petrik@cvut.cz or makarand.tapaswi@iiit.ac.in.

arXiv Source code Download input data

Results - visualization of the optimized trajectories

Pull from left to right Pull from right to left Push from left to right Push from right to left

Pick up Put in front of Put behind Put next to Put onto

Abstract

We aim to teach robots to perform simple object manipulation tasks by watching a single video demonstration. Towards this goal, we propose an optimization approach that outputs a coarse and temporally evolving 3D scene to mimic the action demonstrated in the input video. Similar to previous work, a differentiable renderer ensures perceptual fidelity between the 3D scene and the 2D video. Our key novelty lies in the inclusion of a differentiable approach to solve a set of Ordinary Differential Equations (ODEs) that allows us to approximately model laws of physics such as gravity, friction, and hand-object or object-object interactions. This not only enables us to dramatically improve the quality of estimated hand and object states, but also produces physically admissible trajectories that can be directly translated to a robot without the need for costly reinforcement learning. We evaluate our approach on a 3D reconstruction task that consists of 54 video demonstrations sourced from 9 actions such as pull something from right to left or put something in front of something. Our approach improves over previous state-of-the-art by almost 30%, demonstrating superior quality on especially challenging actions involving physical interactions of two objects such as put something onto something. Finally, we showcase the learned skills on a Franka Emika Panda robot.

Our proposed pipeline is illustrated for four keyframes of a video demonstration corresponding to the action put something next to something. The pipeline consists of: (i) a video preprocessing step which extracts segmentation masks and information about the hand-object contact; (ii) a physics-based optimization step that solves physics equations of motion (ODEs), renders the states via a differentiable renderer and computes and minimizes losses based on perceptual similarity to the video demonstration. In the figure panel, the curves represent trajectories and dots are the discrete timesteps at which loss is computed; and (iii) a re-targeting step which computes the robot trajectory from optimized Cartesian trajectories.

Bibtex

 
@inproceedings{petrik2022real2simphysics,
  author = {Vladimir Petrik and Mohammad Nomaan Qureshi and Josef Sivic and Makarand Tapaswi},
  title = {{Learning Object Manipulation Skills from Video via Approximate Differentiable Physics}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year = {2022}
}

Supplementary video

Acknowledgement

This work was supported by the European Regional Development Fund under the project IMPACT (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000468).