Open-Vocabulary Mobile Manipulation

A benchmark that has a robot find any named object in an unfamiliar house and place it on a specified piece of furniture.

HomeRobot OVMM is a benchmark released in 2023 by Meta FAIR, Georgia Tech, Carnegie Mellon, and Simon Fraser University, and it also served as a NeurIPS 2023 competition. Tasks take the form “move object X from furniture A to furniture B”: the object is specified in text and may belong to a category never seen during training (open vocabulary), and the robot must, in an unfamiliar house, find the object, pick it up, locate the target furniture, and place the object there. The simulated portion is built on Habitat and synthetic HSSD scenes, covering 60 multi-room homes with 2,535 objects across 129 categories; the real-robot portion uses the low-cost Hello Robot Stretch, paired with the open-source HomeRobot software stack. It evaluates navigation, perception, and grasping together as one pipeline, and the paper's baseline achieved a real-robot success rate of about 20%.

ExampleGiven the instruction “move the toy elephant from the chair to the table,” where the toy elephant belongs to a category never seen during training, the robot earns 1 point for each stage it completes — finding the object, picking it up, finding the table, and placing it down — and only counts as fully successful once all four stages are done.

Also called

OVMM, HomeRobot, HomeRobot OVMM, OVMM Challenge

Related

Mobile Manipulation · Open-vocabulary · Habitat · Hello Robot Stretch · Rearrangement · Progress Score

Sources

HomeRobot: Open-Vocabulary Mobile Manipulation (arXiv 2306.11565)
NeurIPS 2023 HomeRobot OVMM Challenge

As of

2023-12