Visual Randomization

Randomly varying a simulation's textures, colors, lighting, and camera during training so a vision policy isn't picky about how things look.

Visual randomization is the part of domain randomization aimed specifically at what the policy sees: object and background textures and colors, light position and intensity, and camera pose are all randomly varied in simulation, with distractor objects added on top, so the model sees enough visual diversity to treat the real world as just one more variation. Tobin and colleagues (OpenAI, Berkeley) did this systematically in 2017, training an object detector using only randomly rendered simulated images and reaching about 1.5 cm localization accuracy in real scenes, which they then used for grasping. It mainly narrows the visual portion of the sim-to-real gap; randomizing physical parameters like mass and friction instead is called dynamics randomization. Isaac Lab has built-in event terms for random textures and random colors that can run automatically at the start of every episode.

ExampleWhen training a vision policy to grasp a cube, each episode might randomly swap the tabletop for wood grain, marble, or a solid color, randomly adjust the light's direction and brightness, and slightly shift the camera position.

Also called

Texture Randomization, Lighting Randomization, Visual Domain Randomization

Related

Domain Randomization · Dynamics Randomization · Sim-to-Real Transfer · Sim-to-Real Gap (Reality Gap) · Visual Generalization · Variant Aggregation (SimplerEnv)

Sources

Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World (arXiv 1703.06907)
Isaac Lab source: envs/mdp/events.py (randomize_visual_texture_material / randomize_visual_color)