Neural Radiance Fields

A neural network that memorizes the color and density of every point in a scene, letting it render any new viewpoint.

Neural radiance fields were proposed by Ben Mildenhall and colleagues at ECCV 2020. A NeRF represents an entire scene with a multi-layer perceptron: given a spatial position (x, y, z) and a viewing direction, it outputs the volume density and color at that point, and volume rendering along each camera ray then synthesizes an image. Training needs only photos with known camera poses, since the rendering process is differentiable — comparing the rendered image against a real photo and minimizing the error is enough to optimize the network. NeRF was originally used for novel view synthesis and later applied to 3D reconstruction and robotics: Dex-NeRF uses it to recover the geometry of transparent objects that depth cameras can't measure accurately, for grasping; LERF and F3RM embed CLIP-like semantic features into a radiance field, making it possible to find objects in a 3D scene using language. Its drawback is that it needs to be optimized separately for every scene and renders slowly; 3D Gaussian splatting, from 2023, renders much faster, and many projects have since switched to it.

ExampleDex-NeRF (CoRL 2021) photographed transparent glassware from multiple angles to train a NeRF, rendered depth from it, and handed that to Dex-Net for grasp planning, reaching 90 to 100 percent grasp success on an ABB YuMi.

Also called

NeRF, Radiance Field

Related

3D Gaussian Splatting · Novel View Synthesis · Distilled Feature Fields · LERF · F3RM · Multilayer Perceptron

Sources

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020)
Dex-NeRF (CoRL 2021, arXiv 2110.14217)
LERF: Language Embedded Radiance Fields (ICCV 2023)