Teleoperation

A human controls a robot in real time while the process is recorded to produce training data.

Teleoperation originally just means operating a machine from a distance — the everyday term “remote control” in academic and technical usage. In embodied AI it is the primary way to collect demonstration data: an operator issues commands through a leader-follower arm, a VR headset, a 3D mouse, or a motion-capture glove, the robot follows along, and the system simultaneously logs camera footage, joint state, and the action taken at every step, producing the “observation-action” pairs that imitation learning needs. Because the data is recorded directly on the robot's own body, the resulting policy can be deployed as-is, with no mismatch between a human hand and a robotic one; the trade-off is that it needs a real robot and a skilled data collector, and is slow and expensive. Handheld capture devices like UMI, and learning from human video, both exist to get around this bottleneck.

ExampleThe ALOHA bimanual platform uses leader-follower teleoperation: an operator moves a pair of leader arms while a matching pair of follower arms tracks them in real time, with the whole hardware setup costing about $20,000. On most tasks, ACT needed just 50 such recorded demonstrations (about 10–20 minutes) to learn fine-grained skills like uncapping a condiment jar or inserting a battery into a remote control.

Also called

teleop, Remote Operation

Related

Demonstration Data · Leader-Follower Teleoperation · VR Teleoperation · Data Collector (Teleoperator) · Imitation Learning · Universal Manipulation Interface

Sources

Wikipedia: Teleoperation
ALOHA / ACT 项目页: Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (Chinese)
Zhao et al. 2023: Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (arXiv 2304.13705)