09Data & Collection

The recorded observations and actions from a human performing a task, used as raw material for robot imitation learning.

One training sample: what the robot 'saw' at a given moment paired with what it 'did' next.

The action value a robot executed at each moment in a dataset — the “answer” imitation learning tries to fit.

Images, depth, joint state, force and touch, language, and other signals recorded in sync while a robot works.

Pressure, shear force, and contact-surface deformation recorded by a tactile sensor when it touches an object.

Aligning readings from cameras, encoders, IMUs, and other sensors onto one shared clock so simultaneous data lines up correctly.

Data recorded from an actual robot physically performing tasks in the real world.

The core bottleneck in embodied AI: usable real robot-interaction data falls far short of what's needed.

Training data generated automatically by having a virtual robot perform tasks inside a physics simulator.

Training data manufactured by computers, via simulation or generative models, instead of collected from the real world.

Video of people doing everyday tasks, collectable without any robot, used to teach robots to understand and copy manipulation.

The vast supply of paired images and text, plus image-based Q&A, collected from the web, teaching a model about the world.

A three-tier framework organizing robot training data by volume and how closely it matches a real robot.

A human controls a robot in real time while the process is recorded to produce training data.

A human physically pushes a robot arm through a motion by hand, and the robot records it as a demonstration.

A human moves a leader arm, and the robot's follower arm copies each joint angle in real time.

Berkeley's open-source low-cost leader-follower teleoperation device, using a scaled-down replica arm to control the real robot.

Teleoperation where position and force are exchanged in both directions between leader and follower, so the operator feels what the robot touches.

The operator wears an exoskeleton matching the robot's joints, controlling it directly with their own arm motion.

A low-cost bimanual exoskeleton from Shanghai Jiao Tong's Cewu Lu group that can both teleoperate a robot and collect data without one.

Controlling a robot arm's end effector in six degrees of freedom by pushing, pulling, twisting, and tilting a 3D mouse.

Controlling a robot in real time by wearing a VR/XR headset and using hand tracking or controllers.

Apple's head-mounted display, whose hand tracking embodied-AI researchers commonly use for teleoperation and collecting human data.

A wearable sensing glove that tracks each finger's bend and the hand's pose in real time.

A teleoperation glove that both records hand motion and pushes force back against the operator's fingers.

A glove covered in pressure or tactile sensors on the fingers and palm, recording the contact forces of a human hand.

Converting a human's (or another robot's) motion into the joint motion a target robot can actually perform.

A general vision-based teleoperation system that uses an ordinary camera to capture hand motion and drive many arms and dexterous hands.

An open-source library that converts human hand keypoint poses into joint angles for many different robotic dexterous hands.

An open-source system for teleoperating a humanoid robot by wearing a VR headset and seeing stereo video from the robot's own viewpoint.

Unitree's open-source XR headset teleoperation program, used to control its humanoid robots and record training data.

A teleoperation system that uses Apple Vision Pro to control two dexterous hands in real time, with vibration feedback, for imitation-learning data collection.

PICO's open-source cross-platform XR teleoperation framework, controlling many kinds of robots with a headset and controllers to collect data.

NVIDIA's open-source teleoperation and data-collection framework, with unified support for XR headsets, gloves, and other input devices.

One operator simultaneously controls a robot's arms, torso, and legs or base — its whole body at once.

Collecting data with no real robot present, by having a human hold or wear a device shaped like the robot's end effector.

A human holds a camera-equipped gripper and performs tasks directly, recording footage and gripper motion as robot training data.

A handheld gripper for recording demonstrations without a real robot, producing data that trains policies deployable on actual arms.

Training a manipulation policy on handheld-gripper human demonstrations, then deploying it on a quadruped fitted with a robot arm.

A hardware-independent handheld-gripper data-collection system and dataset from Shanghai AI Lab, improving on UMI.

AgileX Robotics' handheld data-collection kit that records manipulation demonstrations without needing a real robot.

Having a person wear cameras, gloves, and similar devices while doing their normal job, recording their motion as robot training data.

Stanford's wearable hand motion-capture system that collects data by having a human use their own hands, for teaching robot dexterous hands.

CMU's portable hand-data-collection system that lets ordinary people gather dexterous-manipulation data with their own hands in real-world settings.

A term from China's robotics-data industry: building the capture device to match the robot's end-effector so recorded data transfers directly, with no retargeting.

Sunday Robotics's glove that lets a person do housework wearing it and directly produce robot training data.

A wearable exoskeleton that lets a human hand collect dexterous-hand data directly, then digitally replaces the hand in footage with the robot hand.

MIT's passive hand exoskeleton that lets a human's own hand directly drive a robot hand to collect vision and touch data.

Recording the motion of a person or object into a computer with high precision, using cameras or wearable sensors.

Tracking markers on the body with multiple infrared cameras and triangulating them into 3D motion.

A motion-capture method that straps a set of IMUs to the body to measure each segment's rotation and reconstruct full-body pose.

Estimating a person's 3D motion directly from ordinary video, with no markers or motion-capture suit needed.

Two common motion-capture file formats that store a human skeleton's structure and its joint rotations frame by frame.

A large human-motion library that unifies 15 optical motion-capture datasets into the SMPL body-model format.

Ubisoft La Forge's public human motion-capture dataset, commonly used as a source of motion for humanoid robots to imitate.

A text-to-motion dataset pairing 14,000 clips of 3D human motion with 45,000 natural-language descriptions.

About 10 hours of optical motion-capture data from Stanford, recording full-body motion while carrying everyday objects.

Three common physically-impossible foot errors that appear when human motion is retargeted onto a robot.

Stanford's open-source tool that retargets human motion onto many different humanoid robots in real time.

A data-generation method that retargets a human's interaction with objects and terrain onto a humanoid robot together, not just body pose.

A humanoid-motion dataset built by extracting and captioning motion from massive amounts of internet human videos.

A large-scale humanoid reference-motion dataset, filtered and retargeted under physical constraints.

Video filmed from a head- or glasses-mounted camera, showing the wearer's own point of view with the hands in frame.

Video shot from a camera outside the performer's own body, looking at a person or robot from a bystander's vantage point.

The vast supply of publicly available online video — large and cheap, but without action labels a robot can use.

Video that has only images, no frame-by-frame action record, like human videos and web videos.

Action labels inferred by a model from video, standing in for actions that were never actually recorded.

Editing footage of a human doing a task so it looks like a robot doing it, turning it into usable training data.

Digitally erasing the deployed robot from the camera feed and painting in the robot the policy was trained on, so a vision policy transfers without retraining.

About 220,000 short videos of hands manipulating everyday objects, labeled with 174 action templates.

A classic first-person video dataset of unscripted everyday cooking and kitchen tasks, filmed with head-mounted cameras.

A roughly 3,670-hour egocentric daily-video dataset collected by Meta with 13 universities.

A large-scale video dataset led by Meta, capturing skilled human activities simultaneously from first-person and third-person viewpoints.

Meta's research eyewear for collecting first-person data; it only records, it doesn't display anything to the wearer.

Meta's large-scale first-person dataset of everyday human motion, recorded with Aria glasses and a motion-capture suit.

NVIDIA's multi-view dataset of human hands grasping objects, with 3D pose annotations for both the hand and the object.

A first-person 4D hand-object interaction dataset with 2.4 million RGB-D frames, from Tsinghua and collaborators.

A Shanghai Jiao Tong University hand-object interaction dataset, annotated with object use and two-handed manipulation.

A motion-capture dataset of two-handed dexterous manipulation of articulated objects, with precise 3D meshes for hands and objects every frame.

Meta's first-person hand-object interaction dataset, recorded with Aria glasses and Quest 3, with motion-capture ground truth.

A framework that co-trains a single policy on Project Aria human-hand data and robot data together.

First-person human manipulation data collected with a VR headset, co-trained with humanoid robot data to train one policy.

An Apple dataset of egocentric manipulation video captured with Vision Pro, annotated with precise 3D hand joints.

BeingBeyond's dataset that unifies many kinds of human hand video into hand-motion annotations, for pretraining dexterous-hand VLA models.

Figure's plan to pretrain its humanoid robot using massive amounts of first-person human video.

TARS Robotics's open-source dataset of over a thousand hours of real-world human manipulation, with vision, language, touch, and action.

Build AI's open-source dataset of roughly ten thousand hours of first-person video recorded by factory workers.

Ropedia's open dataset of 10,000 hours of first-person, multimodal human experience, aimed at embodied AI.

A continually growing platform of first-person human demonstration data for robot learning, built jointly by universities and companies.

A standard set of purchasable everyday objects with 3D scans, letting grasping and manipulation research compare results directly.

Google's open-source library of high-precision 3D scans of 1,030 real household objects.

A large-scale library of 3D object meshes organized by WordNet categories, with semantic annotations.

A massive open library of 3D object models, led by the Allen Institute for AI.

A library of articulated 3D object models with joint-motion annotations, ready to load directly into a simulator.

A multisensory object dataset providing an object's appearance, tapping sound, and tactile readings all together.

The most widely used object-detection and segmentation benchmark; LVIS extends COCO's images to over a thousand long-tail categories.

A large-scale indoor RGB-D scan dataset with 3D reconstruction and semantic annotations.

A dataset of RGB-D indoor scans from 90 real buildings, a standard scene library for indoor navigation research.

A dataset of 1,000 real buildings, 3D-reconstructed by Meta and Matterport, used to train navigation agents.

A Shanghai Jiao Tong University dataset of real cluttered-scene grasping data, with over 1.1 billion annotated grasp poses.

An NVIDIA grasp dataset labeled in bulk through physics simulation, containing about 17.7 million parallel-jaw grasps.

A simulation-generated dexterous-hand grasping dataset from Peking University, with 1.32 million grasps across 5,355 objects.

Demonstration data generated automatically by a hand-written rule-based program controlling the robot, instead of a human.

A system that segments a few human demonstrations by object, transforms and stitches them, and auto-generates many new demonstrations.

NVIDIA's automatic data-generation system that expands a few dozen demonstrations into tens of thousands of bimanual dexterous-hand demonstrations.

A method that automatically turns one real-robot demonstration into many synthetic demonstrations with objects in new positions.

A generative simulation framework where a large model proposes tasks, builds the simulated scene, and learns the skill.

NVIDIA's open collection of robotics and autonomous-driving datasets hosted on Hugging Face.

A billion-frame simulated synthetic grasping dataset built by Galbot and others, used to pretrain GraspVLA.

A large-scale simulated synthetic dataset from Shanghai AI Lab, built for pretraining general-purpose robot policies.

Using image or video generation models to rewrite existing robot data, producing new-scene training examples from old demonstrations.

Synthetic robot training data generated by a video world model, with pseudo action labels added afterward.

A method that renders large amounts of robot training data from a phone scan plus one video of a human demonstration.

Data collected from many different kinds of robots and pooled together for training.

A Google-led open dataset pooling more than a million real-robot trajectories from 22 kinds of robots.

Robot data from different robots, sensors, and collection methods, with mismatched formats and meanings.

A 2019 dataset of multi-robot interaction video, about 15 million frames across 7 kinds of robots.

A 2021 Google dataset of real-robot demonstrations across 100 tasks, collected to study zero-shot task generalization.

Google's tabletop block-pushing dataset, with nearly 600,000 trajectories paired with natural-language instructions.

Google's real-robot manipulation dataset, collected over 17 months using 13 robots.

A multimodal real-robot manipulation dataset from Shanghai Jiao Tong University, with force, audio, and human demonstration video.

A roughly 60,000-trajectory multi-task manipulation dataset collected by UC Berkeley on a low-cost WidowX arm.

A real-robot kitchen-task dataset from CMU and Meta, with multi-task demonstrations and 4 camera views per frame.

A real-robot manipulation dataset collected by 50 people across 564 scenes on three continents using Franka arms.

A unified embodied-data format standard from Peng Cheng Laboratory and partners, and a roughly 3-million-item dataset built to that standard.

A multi-embodiment real-robot manipulation dataset from the Beijing Humanoid Robot Innovation Center and Peking University, collected under a unified protocol.

A large-scale real-robot manipulation dataset from AgiBot and partners, with the Beta release exceeding a million trajectories.

Fourier's open-source teleoperated dataset of about 30,000 humanoid-plus-dexterous-hand manipulation trajectories.

A roughly 500-hour dataset Galaxea collected at real-world locations using a single model of mobile bimanual robot.

A real-robot humanoid manipulation dataset from USC and the Toyota Research Institute, covering 260 tasks.

A cross-embodiment bimanual manipulation dataset led by BAAI, open-sourced across 15 kinds of robots.

Ai2's open-source bimanual teleoperation dataset for MolmoAct2, totaling more than 720 hours.

A video question-answering dataset from Google DeepMind, built around long-horizon robot tasks.

A cross-embodiment visuotactile dataset released in 2026 by China's National Humanoid Robot Innovation Center and Weitai Robotics.

A large-scale real-robot manipulation dataset with vision-and-touch signals, released by Daimon Robotics in 2026.

A general-purpose file format that stores many large arrays hierarchically in one file, commonly used for robot demonstration data.

An open-source format for storing large multidimensional arrays in compressed chunks; Diffusion Policy and UMI use it for training data.

A Google-proposed standard format and toolchain for organizing sequential-decision data as episodes made of steps.

Google's open-source dataset library that downloads and loads a dataset into a trainable pipeline in one line of code.

TensorFlow's binary file format, storing a sequence of serialized records back to back.

LeRobot's standard dataset format, storing state and actions in tables and camera footage as video.

An open-source column-oriented file format; LeRobot datasets use it to store numeric data like state and actions.

A format that packages training samples into same-named files inside a series of tar shards, for fast large-scale sequential reads.

An open-source multimodal logging file format from Foxglove, the default recording format for ROS 2 bags.

Finding and fixing or removing bad data before training, such as failed trajectories, empty actions, or misaligned frames.

Removing timesteps where the robot doesn't move from demonstrations before training, so the model doesn't learn to freeze in place.

Checking each piece of collected robot data before training to catch and discard unusable or flawed samples.

Re-sending a recorded action sequence to a robot or simulator to check whether the data can be reproduced.

Blurring or removing personal information like faces, license plates, and voices from data so specific individuals can't be identified.

Adding text descriptions, segment boundaries, bounding boxes, and other labels to raw data to tell a model what it is.

Attaching a natural-language description to a robot trajectory, such as 'put the red cup in the sink,' so a model can follow instructions.

Splitting one long demonstration into steps, marking each segment's start and end frame with a description.

Using pretrained models or scripts, instead of humans, to automatically add language instructions, bounding boxes, and other labels to robot data.

Robot data recorded while an operator freely manipulates a scene with no specific task in mind.

After data is collected, relabeling a trajectory's goal or instruction to match whatever it actually achieved.

Automatically writing or rewriting language instructions for robot trajectories, so the same motion gets many phrasings.

How broadly training data varies across scenes, objects, tasks, and viewpoints.

Research into how, in robot imitation learning, generalization improves as training data grows.

Data collected in homes, offices, outdoors, and other uncontrolled real-world settings, rather than a fixed lab bench.

Human demonstration data that includes wasted motion, hesitation, mistakes, or inconsistent quality across operators.

Picking out the subset of a large robot dataset that actually helps training, and discarding the harmful part.

Using a handful of demonstrations for a target task to search a large dataset for similar clips, then training on the combined set.

The proportion each data source is given when training on a mix of multiple sources at once.

The recipe Octo and OpenVLA used to select and weight subsets of Open X-Embodiment for pretraining.

When information that should only appear at test time leaks into training, inflating evaluation scores beyond real-world performance.

A facility built to replicate real-world settings, where large numbers of robots are teleoperated to mass-produce training data.

A front-line worker who records robot training data through teleoperation or wearable demonstration devices.

The written procedure that specifies exactly how each demonstration should be collected and what counts as an acceptable one.

The portion of raw collected data that passes quality control and can actually be used to train a model.

Distributing robot data-collection tasks to large numbers of non-professional workers, who are paid for each validated, usable clip.

Stanford's 2018 platform for crowdsourced robot demonstrations, teleoperated remotely from a smartphone.

Letting a robot propose its own tasks, attempt them, and log the results, with little or no human teleoperation.

Trajectories that didn't complete the task, useful for learning to recover from mistakes and for training reward models.

Data recorded when a human takes over from an autonomously running policy just as it's about to make a mistake.

Demonstration data specifically recorded to show how a robot gets back on track after starting to go wrong.

A term from China's robotics industry: feeding data generated during real-world robot deployment back into training, then redeploying the updated model.

A self-reinforcing loop: deployment produces data, the data improves the model, and the better model gets deployed even more widely.