In-hand Manipulation

Using the fingers to adjust an object's orientation or position while still holding it, without setting it down to regrasp.

In-hand manipulation means a robot with a multi-fingered hand rotates, translates, or reorients an object it is holding using coordinated finger motion, without ever releasing it; the typical task is turning an object to a specified orientation, called in-hand reorientation. Spinning a pen, turning a key, and solving a Rubik's cube all rely on this ability in humans. The difficulty is that contact points are numerous and constantly changing — fingers must repeatedly make and break contact with the object, called finger gaiting — and the hand's own body often blocks the view of what it is doing. A landmark example is OpenAI's 2018 “Learning Dexterous In-Hand Manipulation”: a Shadow dexterous hand was trained in simulation with reinforcement learning to reorient a block, while friction coefficients, object appearance, and other physical and visual properties were randomized (domain randomization), then transferred directly to the real hand; behaviors such as finger gaiting, multi-finger coordination, and using gravity emerged naturally during training. It is one of the hardest sub-problems within dexterous manipulation, and often comes up together with tactile sensing and sim-to-real transfer.

ExampleOpenAI's Shadow dexterous hand, using only finger movement, rotates a lettered block resting in its palm until the specified face points up (the Dactyl project).

Also called

In-hand Reorientation

Related

Dexterous Manipulation · Dexterous Hand · Finger Gaiting · Dactyl · Domain Randomization · Tactile Sensor

Sources

Learning Dexterous In-Hand Manipulation (OpenAI, 2018)