Real-World Humanoid Locomotion with Reinforcement Learning

A Transformer walking controller trained purely with reinforcement learning in simulation, deployed zero-shot to make a humanoid walk outdoors.

This work was released by Ilija Radosavovic, Jitendra Malik, and colleagues at UC Berkeley in March 2023 and published in Science Robotics in 2024. Humanoid walking had mostly relied on classical controllers, which struggle to adapt to new environments. This paper uses a causal Transformer as the controller: it takes in a window of past proprioceptive observations and actions and outputs the next action, adapting to terrain 'in context' from its history without updating any parameters. Training happens entirely in Isaac Gym simulation: a teacher policy is first trained with privileged information (state information unavailable on the real robot), then a student policy is trained by combining imitation of the teacher with reinforcement learning, together with domain randomization, before being deployed zero-shot to Agility Robotics' Digit humanoid. The resulting controller walks across a range of outdoor terrains and resists pushes, making this an early landmark for learning-based humanoid locomotion control.

ExampleDigit walks on outdoor grass and rubberized running tracks it never saw during training, and stays stable even when pushed from the side by a person.

Also called

Real-World Humanoid Locomotion with RL, Radosavovic et al., Science Robotics 2024

Related

RL-based Locomotion Control · Sim-to-Real Transfer · Teacher-Student Distillation · Domain Randomization · Agility Robotics Digit · Humanoid Locomotion as Next Token Prediction

Sources

Real-World Humanoid Locomotion with Reinforcement Learning (arXiv 2303.03381)
Project page

As of

2024