ProcTHOR (Large-Scale Embodied AI Using Procedural Generation)

Allen Institute for AI's framework for procedurally generating large numbers of interactive 3D houses for embodied-AI training.

ProcTHOR was released by the Allen Institute for AI (Ai2) in June 2022 and won an Outstanding Paper award at NeurIPS 2022; it is built on top of the AI2-THOR simulator. Before it, embodied agents could only train in a few dozen to a few hundred hand-built scenes, which made them prone to memorizing the map and failing in a new house. ProcTHOR instead uses procedural generation (building content automatically from rules plus random sampling) to mass-produce houses: it first decides room types and counts, then generates the floor plan and places furniture and small objects, randomizing materials and lighting as it goes, drawing from a library of 108 categories and 1,633 interactive objects. The team publicly released ProcTHOR-10K, a set of 10,000 houses. Agents pretrained purely on RGB images inside these houses achieved then-state-of-the-art results on 6 benchmarks — object navigation, object rearrangement, and arm-pointing navigation among them — and performed strongly zero-shot (with no fine-tuning) on some tasks. Later navigation models such as PoliFormer were also trained with reinforcement learning inside its generated houses.

ExampleSpecify “two bedrooms plus a kitchen and living room,” and ProcTHOR automatically generates the floor plan, furnishes it with a bed, a sofa, and a refrigerator, and randomizes the flooring material and lighting — producing a new house an agent can enter, open the fridge in, and search for an apple.

Also called

ProcTHOR-10K

Related

AI2-THOR · Procedural Generation · Object-Goal Navigation · Rearrangement · PoliFormer · Holodeck

Sources

ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (arXiv 2206.06994)
ProcTHOR 项目主页 (Chinese)
NeurIPS 2022 Awards

As of

2024-06