XRoboToolkit

PICO's open-source cross-platform XR teleoperation framework, controlling many kinds of robots with a headset and controllers to collect data.

XRoboToolkit was developed by a team at PICO, the XR company owned by ByteDance (Zhigen Zhao, Liuchuan Yu, Ke Jing, and Ning Yang), with the paper posted to arXiv in July 2025, and won the Best Paper award at SII 2026. It targets the large amount of demonstration data VLA models need: on the headset side, it follows the OpenXR standard (a cross-vendor interface specification for XR devices), supporting head, controller, and hand tracking plus additional motion trackers (such as one strapped to the elbow to control elbow position); on the robot side, it provides Python and C++ interfaces, uses optimization-based inverse kinematics to convert the operator's hand pose into joint commands, and streams the robot's stereo camera view back to the headset with low latency. The framework is modular and has already been integrated with MuJoCo simulation, dual UR5 arms, mobile robots, and dexterous hands. The paper used data collected with it to train a VLA, validating its effectiveness on fine manipulation tasks.

ExampleAn operator wears a PICO 4 Ultra headset and uses two controllers to simultaneously control dual UR5 arms performing a precise connector-insertion task, seeing stereo video streamed back from the robot's cameras inside the headset.

Also called

XRoboToolkit: A Cross-Platform Framework for Robot Teleoperation

Related

Teleoperation · VR Teleoperation · Unitree xr_teleoperate · Meta Quest 3 / PICO 4 Ultra XR Headsets · Inverse Kinematics (IK) · Demonstration Data

Sources

arXiv 2508.00097: XRoboToolkit
XRoboToolkit 项目主页 (Chinese)
XR-Robotics/XRoboToolkit-Teleop-Sample-Python (GitHub)

As of

2026-09