H2O

A 2024 CMU humanoid teleoperation framework that uses a single RGB camera to make a robot mimic a person's whole-body motion in real time.

H2O was released in March 2024 by teams under Guanya Shi, Changliu Liu, and Kris Kitani at Carnegie Mellon University, published at IROS 2024 as an oral presentation, with Tairan He and Zhengyi Luo as co-first authors. It uses nothing but an ordinary RGB camera: the pose estimator HybrIK computes human body pose from the footage in real time, and a whole-body tracking policy trained with reinforcement learning drives a Unitree H1 to follow along. The key step is “sim-to-data”: about 10,000 motion clips from the AMASS motion-capture dataset are first retargeted onto the H1, then tried out in simulation by an imitation policy with access to privileged information, filtering out motions the robot can't achieve, leaving about 8,500 clips for training; the trained policy is deployed to the real robot with no extra tuning. The authors call this the first learning-based real-time whole-body humanoid teleoperation. Follow-up work, OmniH2O, extended it to VR teleoperation and autonomous learning from teleoperation data.

ExampleThe operator stands in front of the camera and walks, kicks, turns, waves, and throws punches, and the H1 follows along with the same motions in real time; a backflip was also demonstrated.

Also called

Human to Humanoid, Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation

Related

Whole-Body Teleoperation · Motion Retargeting · Motion Tracking · OmniH2O · AMASS (Archive of Motion Capture as Surface Shapes) · Unitree H1

Sources

Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation (arXiv 2403.04436)
H2O 项目页 (Chinese)

As of

2024-10