RoboCat

DeepMind's cross-embodiment manipulation agent, built on Gato, that generates its own data to keep improving.

RoboCat is a robot manipulation agent released by Google DeepMind in June 2023, reusing the architecture of Gato, DeepMind's multimodal generalist model: it is a decision Transformer conditioned on a goal image — given a picture of 'what the task looks like when done,' it outputs actions. It trains on millions of trajectories from a variety of real and simulated robot arms, and can handle robots whose observation and action formats differ from one another. Its defining feature is a 'self-improvement' loop: a task-specific version is first fine-tuned on 100 to 1,000 human demonstrations, then lets the robot practice roughly 10,000 times on its own to generate new data, which is folded back into the training set to retrain the generalist model. According to DeepMind's blog, later versions raised the success rate on new tasks from 36% to 74%. RoboCat is an early landmark demonstrating that cross-embodiment data can speed up learning new skills.

ExampleRoboCat learned to operate a new robot arm fitted with a three-fingered gripper within a few hours, reaching an 86% success rate at grasping a gear after just 1,000 demonstrations.

Also called

RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation

Related

Gato · Decision Transformer · Cross-Embodiment · Self-improvement · Goal-conditioned Policy · Google DeepMind

Sources

arXiv 2306.11706: RoboCat
Google DeepMind 博客：RoboCat (Chinese)

As of

2023-12