VR Teleoperation

Controlling a robot in real time by wearing a VR/XR headset and using hand tracking or controllers.

VR teleoperation is a form of teleoperation: the operator wears a VR/XR headset, which displays the video streamed back from the robot's camera; the headset's hand tracking or controllers read out the human's wrist and finger poses, which are then converted into robot commands through inverse kinematics and motion retargeting (translating human hand pose into robot joint angles). It is cheaper and more portable than a leader-follower arm, and suits humanoid robots and dexterous hands, which have many more degrees of freedom — datasets such as BridgeData V2, DROID, and AgiBot World have all used it. Its downside is that it usually has no force feedback, and hand tracking has latency and jitter, making it less stable than a leader-follower arm for fine-contact tasks; AgiBot World's paper also notes that its VR setup could only produce a few preset dexterous-hand gestures, switching to motion capture for more complex tasks.

ExampleOpen-TeleVision uses an Apple Vision Pro to control a Unitree H1: the robot's head follows the operator's head movements, its stereo camera feed streams back to the headset, and both hands follow via dex-retargeting; Unitree's open-source xr_teleoperate supports Apple Vision Pro, PICO 4 Ultra Enterprise, and Meta Quest 3, and saves each episode for imitation learning.

Also called

XR Teleoperation, Headset Teleoperation

Related

Teleoperation · Open-TeleVision · Apple Vision Pro · Motion Retargeting · Unitree xr_teleoperate · VR Headset

Sources

Cheng et al. 2024: Open-TeleVision (arXiv 2407.01512)
GitHub: unitreerobotics/xr_teleoperate
AgiBot World Colosseo 技术报告 (arXiv 2503.06669) (Chinese)

As of

2026-09