Human-Gated DAgger

A DAgger variant where a human takes over just before the robot is about to err, and only that takeover data gets used for retraining.

Introduced by Stanford's Kelly and colleagues in 2018. The original DAgger (Dataset Aggregation) requires an expert to state, step by step, what action should be taken while the policy is actually driving the system — but it's hard for a person to give accurate labels, and unsafe, when they aren't the one actually in control of the car or robot. HG-DAgger instead lets a human decide when to intervene: the learned policy is normally in control, and the human takes over the moment they see it heading somewhere dangerous, steers the system back to a safe state, and then hands control back; only the observation-action pairs recorded during that takeover get added to the dataset, and the policy is retrained on the aggregated data. The paper also uses disagreement among an ensemble of networks to estimate the policy's uncertainty, and learns a risk threshold from the moments of human intervention. Experiments ran in both simulation and on a real self-driving car. This pattern of human takeover plus collecting correction data is now a common human-in-the-loop workflow in real-robot post-training.

ExampleA self-driving policy is running on a test vehicle; just before it drifts over the lane line, the safety driver takes the wheel and pulls the car back to the center of the lane, and this correction data gets added to the training set.

Also called

HG-DAgger

Related

DAgger · Human-in-the-Loop · Compounding Error · Human Intervention Data · Recovery and Correction Data · Interactive Imitation Learning

Sources

HG-DAgger: Interactive Imitation Learning with Human Experts (arXiv 1810.02890)