Residual Policy

Learning a correction on top of an existing controller or policy's output, and adding the two together as the final action.

A residual policy doesn't learn an entire policy from scratch; instead, it keeps an existing base policy (a hand-designed controller, model predictive control, or an already-trained imitation-learning policy) and trains a separate network that only outputs a correction to it, with the final action equal to the base action plus the residual. MIT's Silver and colleagues proposed residual policy learning in 2018, and around the same time Johannink, Levine, and colleagues validated residual reinforcement learning on real-robot block assembly. The benefit is that the base policy can already do the task roughly right, so the residual only has to fill in the parts that are hard to model, such as friction, contact, and calibration error, which means a small exploration space, good sample efficiency, and more safety. A common recent pattern is to freeze a diffusion policy or VLA and train a lightweight, closed-loop, step-by-step residual with reinforcement learning to boost success rate on tasks like fine assembly.

ExampleResiP (Pulkit Agrawal's group, 2024) freezes a diffusion policy that uses action chunking, trains a step-by-step closed-loop residual policy with PPO to correct it, substantially raises success rate on simulated tasks like FurnitureBench furniture assembly, and then transfers it to the real robot via teacher-student distillation.

Also called

Residual Policy Learning, RPL

Related

Residual Reinforcement Learning · Diffusion Policy · Reinforcement Fine-Tuning (RL Fine-Tuning) · Action Chunking · Model Predictive Control · Teacher-Student Distillation

Sources

Residual Policy Learning (Silver et al., arXiv 1812.06298)
Residual Reinforcement Learning for Robot Control (Johannink et al., arXiv 1812.03201)
From Imitation to Refinement -- Residual RL for Precise Assembly (ResiP, arXiv 2407.16677)

As of

2024-07