RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator

A robot simulator that reconstructs real scenes with 3D Gaussian Splatting and couples them to a physics engine.

RoboGSim is a paper and system released in November 2024, with authors from the Harbin Institute of Technology (Shenzhen), the Institute of Computing Technology at the Chinese Academy of Sciences, Megvii, and Zhejiang University. It combines 3D Gaussian Splatting (a 3D representation that reconstructs a scene as a large number of colored ellipsoids and renders results close to photographs) with the Isaac Sim physics engine, and is made up of four parts: a Gaussian reconstructor, a digital-twin builder, a scene composer, and an interactive engine. It first reconstructs a real tabletop scene into simulation, then inside simulation swaps camera viewpoints, objects, trajectories, and backgrounds to synthesize new data, and finally deploys back to a real robot — that is, Real2Sim2Real. The paper reports that a policy trained purely on data synthesized this way can run zero-shot on a real robot, matching the performance of a policy trained on real-robot data, and doing better still under new viewpoints and new scenes. It can also serve as a closed-loop evaluation environment for testing VLA models.

ExampleFirst film a lab bench with a camera, reconstruct the tabletop and robot arm inside RoboGSim, then swap in new camera viewpoints to batch-generate pick-and-place trajectories, and use that synthetic data to train a policy that deploys directly to the real arm.

Related

Gaussian Splatting-based Simulation · Real-to-Sim-to-Real · 3D Gaussian Splatting · Digital Twin · Sim-to-Real Transfer · NVIDIA Isaac Sim

Sources

arXiv 2411.11839 - RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator
RoboGSim 项目主页 (Chinese)

As of

2025-08