SAM-6D

A method that uses SAM’s segmentation to estimate the 6D pose of objects it was never trained on, given only a CAD model.

SAM-6D is a zero-shot 6D object pose estimation method proposed in 2023 by Jiehong Lin, Kui Jia, and colleagues, from institutions including the Chinese University of Hong Kong, Shenzhen, and South China University of Technology, published at CVPR 2024. 6D pose means an object’s 3D position plus its 3D orientation, essential for both grasping and assembly; “zero-shot” means that for a new object never seen during training, the method can estimate its pose given only a CAD model, with no retraining required. The method works in two steps: first, SAM generates all candidate regions, which are scored and filtered by semantics, appearance, and geometry to find the target object; then pose estimation is framed as partial-to-partial point matching between the object’s model point cloud and the observed point cloud, solved in a coarse-to-fine two-stage process. Inputs are an RGB-D image, camera intrinsics, and a CAD model; the paper reports results exceeding prior methods across all 7 core datasets of the BOP benchmark.

ExampleGiven a CAD model of a part, SAM-6D locates that part in an RGB-D image of a cluttered parts bin and outputs its 6D pose for a robot arm to grasp.

Also called

Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation

Related

6D Object Pose Estimation · Segment Anything Model · FoundationPose · MegaPose · BOP (Benchmark for 6D Object Pose Estimation) · Point Cloud Registration

Sources

arXiv 2311.15707: SAM-6D
JiehongLin/SAM-6D GitHub

As of

2024-06