3D Scene Graph

Organizes a 3D scene’s floors, rooms, objects, and their relationships into a layered graph.

A 3D scene graph is a structured scene representation: nodes are entities in the scene — buildings, floors, rooms, objects, camera positions, and so on — and edges are the relationships between them, like “inside,” “on top of,” or “next to,” with each node carrying attributes such as category, size, and 3D position. Armeni and colleagues introduced this structure at ICCV 2019, with building, room, object, and camera layers; MIT’s Hydra (RSS 2022) lets a robot build a hierarchical scene graph in real time while exploring. Compared with a dense point cloud or mesh, a scene graph is compact and semantically clear — it can directly answer “which room is the fridge in?” — and it’s easy to convert to text for a large language model to use in task planning. ConceptGraphs builds open-vocabulary scene graphs using 2D foundation models, while SayPlan has a large model search and plan long-horizon, multi-room tasks directly over a scene graph.

ExampleA robot given the instruction “bring the apple from the kitchen table to the living room” first follows kitchen → table → apple through the scene graph to find the target node and the living-room node, then calls its navigation, grasping, and placing skills in sequence.

Also called

3DSG

Related

ConceptGraphs · SayPlan · Semantic Map · Scene Understanding · LLM-based Task Planning

Sources

3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera (arXiv 1910.02527)
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization (arXiv 2201.13360)
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (arXiv 2309.16650)