UMI on Legs

Training a manipulation policy on handheld-gripper human demonstrations, then deploying it on a quadruped fitted with a robot arm.

UMI on Legs is a CoRL 2024 paper from July 2024 by Shuran Song's group at Stanford and Columbia University (Huy Ha, Yihuai Gao, and others). The hardware is a Unitree Go2 quadruped carrying a 6-axis ARX5 robot arm on its back. The approach splits into two halves: the manipulation skill is trained on demonstrations collected in the real world with a UMI handheld gripper (a gripper fitted with a GoPro, which needs no robot to collect data), producing a diffusion policy; coordinating the legs and arm is handled by a whole-body controller (a low-level controller driving both legs and arm together) trained with reinforcement learning in simulation. Only the “end-effector trajectory in task coordinates” is passed between the two halves, so a policy trained for a fixed-base robot arm can be mounted directly onto the quadruped and used as-is. The paper reports over 70% success across three task categories: throwing a ball, pushing a kettlebell, and arranging cups.

ExampleFor the quadruped to throw a ball into a bucket: the upper-level policy, trained on UMI data, outputs the gripper's target trajectory, while the whole-body controller adjusts the legs' posture, keeps balance, and carries out the throwing motion.

Also called

UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers

Related

Universal Manipulation Interface · Handheld Gripper Data Collection · Whole-Body Control · Diffusion Policy · Mobile Manipulation · Quadruped Robot

Sources

UMI on Legs 项目主页 (Chinese)
arXiv 2407.10353: UMI on Legs

As of

2024-07