Visually Guided Model Predictive Robot Control via
6D Object Pose Localization and Tracking

M. Fourmy

V. Priban

J. K. Behrens

N. Mansard

J. Sivic

V. Petrík

[Paper]

[Video]

Abstract

The objective of this work is to enable manipulation tasks with respect to the 6D pose of a dynamically moving object using a camera mounted on a robot. Examples include maintaining a constant relative 6D pose of the robot arm with respect to the object, grasping the dynamically moving object, or co-manipulating the object together with a human. Fast and accurate 6D pose estimation is crucial to achieve smooth and stable robot control in such situations. The contributions of this work are three fold. First, we propose a new visual perception module that asynchronously combines accurate learning-based 6D object pose localizer and a high- rate model-based 6D pose tracker. The outcome is a low-latency accurate and temporally consistent 6D object pose estimation from the input video stream at up to 120 Hz. Second, we develop a visually guided robot arm controller that combines the new visual perception module with a torque-based model predictive control algorithm. Asynchronous combination of the visual and robot proprioception signals at their corresponding frequencies results in stable and robust 6D object pose guided robot arm control. Third, we experimentally validate the proposed approach on a challenging 6D pose estimation benchmark and demonstrate 6D object pose-guided control with dynamically moving objects on a real 7 DoF Franka Emika Panda robot.

Supplementary video

Paper and Supplementary Material

Mederic Fourmy, Vojtech Priban, Jan Kristof Behrens, Nicolas Mansard, Josef Sivic, Vladimir Petrik
Visually Guided Model Predictive Robot Control via 6D Object Pose Localization and Tracking

(hosted on ArXiv)


@misc{fourmy2023visually,
    title={Visually Guided Model Predictive Robot Control via 6D Object Pose Localization and Tracking}, 
    author={Mederic Fourmy and Vojtech Priban and Jan Kristof Behrens and Nicolas Mansard and Josef Sivic and Vladimir Petrik},
    year={2023},
    eprint={2311.05344},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

[Bibtex]

Acknowledgements

This work was partly supported by the AGIMUS project, funded by the European Union under GA no.101070165, by the European Re- gional Development Fund under project Robotics for Industry 4.0 (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000470), and by the Czech Science Foundation (project no. GA21-31000S). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them.