Rendering

The process of computing a camera image from a 3D scene; it produces all of a simulator's camera-based sensor data.

Rendering is a computer-graphics term for computing a 2D image from 3D models, materials, lighting, and camera parameters. The main methods are rasterization (projecting triangles onto the screen and shading pixel by pixel, which is fast and used by games and most real-time simulators) and ray tracing or path tracing (simulating how light actually travels, which is more realistic but slower). In robot simulation, the physics engine handles how objects move while the renderer handles “taking the picture”: producing RGB images, depth maps, and segmentation masks as camera observations for training and evaluating vision policies, as well as for visualizations meant for humans to watch. Rendering quality determines the size of the visual sim-to-real gap, while rendering speed is often the bottleneck for vision-based reinforcement learning, which is why ManiSkill3, Isaac Lab, and others all implement batched GPU rendering. Running a simulation without opening a graphical window is called headless mode.

ExampleManiSkill3 runs both physics simulation and rendering in parallel on the GPU; its paper reports that simulation with rendering reaches over 30,000 frames per second on its benchmark environments, 10 to 1000 times faster than other platforms and using 2 to 3 times less GPU memory.

Also called

Rendering Engine, Renderer

Related

Rasterization · Ray Tracing · Path Tracing · Photorealistic Rendering · Batched Rendering · Headless Mode

Sources

Wikipedia: Rendering (computer graphics)
ManiSkill3: GPU Parallelized Robotics Simulation and Rendering (arXiv 2410.00425)