Universal Manipulation Interface

A handheld gripper for recording demonstrations without a real robot, producing data that trains policies deployable on actual arms.

UMI was proposed in February 2024 by researchers at Stanford, Columbia, and the Toyota Research Institute (Cheng Chi, Shuran Song, and colleagues). A person holds a 3D-printed parallel gripper and performs tasks directly; the gripper carries a 155° fisheye GoPro plus small mirrors on either side for extra viewpoints. Afterward, the visual SLAM algorithm ORB-SLAM3, combined with the GoPro's built-in IMU (inertial measurement unit), recovers the gripper's 6-degree-of-freedom pose in space, while marker tracking recovers how far the fingers opened, together yielding action labels. Because the handheld gripper matches the position of the gripper and camera mounted on a real robot, collection needs no robot present at all, so data can be gathered at home, outdoors, or anywhere in the real world. Paired with latency matching (compensating for the different camera and motor delays across robots) and a relative-trajectory action representation, the same policy can be deployed on both a UR5e and a Franka.

ExampleThe paper trained policies on UMI data that could set a cup down, wash dishes, toss objects into matching bins, and fold clothes with two arms. The gripper hardware costs about $73, the GoPro and accessories about $298, and on the cup-placing task it collected about 111 demonstrations per hour — more than 3 times faster than SpaceMouse (3D mouse) teleoperation.

Also called

UMI, UMI Gripper, Handheld UMI Collection

Related

Handheld Gripper Data Collection · Robot-free (Embodiment-free) Data Collection · FastUMI · UMI on Legs · DexUMI · RDT2

Sources

Chi et al. 2024: Universal Manipulation Interface (arXiv 2402.10329)
UMI 项目页 (Chinese)

As of

2024-03