RoboSet

A real-robot kitchen-task dataset from CMU and Meta, with multi-task demonstrations and 4 camera views per frame.

RoboSet is a real-robot dataset that Carnegie Mellon University and Meta AI made public as part of the RoboAgent project (2023), set mostly in home-kitchen activities. According to the dataset's page, it totals 28,500 trajectories: 9,500 collected through teleoperation with an Oculus Quest 2 controller, and 19,000 obtained through kinesthetic-teaching replay (replaying an existing trajectory in a rearranged version of the scene). Every frame carries 4 camera views, and tasks are organized by “activity,” with each activity containing 4–6 language-instructed subtasks. RoboAgent used about 7,500 of these trajectories to train its MT-ACT policy, which learned 12 manipulation skills. The data is released in HDF5 format under the MIT license.

ExampleOpening one RoboSet trajectory with h5py gives images from 4 camera views, the robot arm's state, the corresponding actions, and the language instruction for that task.

Related

RoboAgent · Action Chunking · Teleoperation · Kinesthetic Teaching · Multi-View · Hierarchical Data Format version 5

Sources

RoboSet 数据集页面 (Chinese)
RoboAgent 项目主页 (Chinese)

As of

2023-09