The objective of this work is to enable manipulation tasks with respect to the 6D pose of a dynamically
moving object using a camera mounted on a robot. Examples
include maintaining a constant relative 6D pose of the robot
arm with respect to the object, grasping the dynamically
moving object, or co-manipulating the object together with
a human. Fast and accurate 6D pose estimation is crucial to
achieve smooth and stable robot control in such situations. The
contributions of this work are three fold. First, we propose a
new visual perception module that asynchronously combines
accurate learning-based 6D object pose localizer and a high-
rate model-based 6D pose tracker. The outcome is a low-latency
accurate and temporally consistent 6D object pose estimation
from the input video stream at up to 120 Hz. Second, we develop
a visually guided robot arm controller that combines the new
visual perception module with a torque-based model predictive
control algorithm. Asynchronous combination of the visual and
robot proprioception signals at their corresponding frequencies
results in stable and robust 6D object pose guided robot
arm control. Third, we experimentally validate the proposed
approach on a challenging 6D pose estimation benchmark and
demonstrate 6D object pose-guided control with dynamically
moving objects on a real 7 DoF Franka Emika Panda robot.
Supplementary video
Paper and Supplementary Material
Mederic Fourmy, Vojtech Priban, Jan Kristof Behrens, Nicolas Mansard, Josef Sivic, Vladimir Petrik Visually Guided Model Predictive Robot Control via 6D Object Pose Localization and Tracking
@misc{fourmy2023visually,
title={Visually Guided Model Predictive Robot Control via 6D Object Pose Localization and Tracking},
author={Mederic Fourmy and Vojtech Priban and Jan Kristof Behrens and Nicolas Mansard and Josef Sivic and Vladimir Petrik},
year={2023},
eprint={2311.05344},
archivePrefix={arXiv},
primaryClass={cs.RO}
}
This work was partly supported by the AGIMUS project, funded by
the European Union under GA no.101070165, by the European Re-
gional Development Fund under project Robotics for Industry 4.0 (reg.
no. CZ.02.1.01/0.0/0.0/15 003/0000470), and by the Czech Science
Foundation (project no. GA21-31000S). Views and opinions expressed are
however those of the author(s) only and do not necessarily reflect those of
the European Union or the European Commission. Neither the European
Union nor the European Commission can be held responsible for them.
