Task Generalization

A policy's ability to complete a new task or new instruction that never appeared during training.

This dimension of generalization asks whether a model can perform a task absent from its training data — for instance, having only learned “put the apple in the bowl” and “open the drawer,” can it complete “put the apple in the drawer”? It is usually harder than switching objects or scenes, since it requires recombining learned actions in a new way and understanding new semantics. Google's BC-Z (2022), after training on more than 100 tasks, reached an average 44% success rate on 24 entirely new tasks with zero demonstrations; RT-Trajectory (2023) points out that a policy conditioned only on language struggles to transfer from pick-and-place to a motion as different as folding, and instead conditions on a rough sketched trajectory. VLA models, drawing on a large model's semantic knowledge, are also hoped to improve this kind of generalization.

ExampleA robot arm that has only ever learned “put the apple in the bowl” and “open the drawer” is asked to “put the apple in the drawer” — a task combination absent from its training data; succeeding at it demonstrates task generalization.

Also called

Cross-task Generalization

Related

Generalization · Zero-shot · Compositional Generalization · Semantic Generalization · RT-Trajectory · Instruction Following

Sources

BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches