Tool Use

A robot using an external object as a tool to complete a task it cannot do bare-handed.

Tool use means a robot treats an object in its environment as an extension of its arm — pulling a distant item closer with a hook, stir-frying with a spatula, tightening a screw with a screwdriver. The difficulty lies in understanding a tool's shape and affordance (how an object can be used), grasping the contact mechanics between the tool and the target object, and replanning motion once the tool is in hand. A 2023 study, RoboTool, splits “creative tool use” into three categories: selecting the right tool from several objects, using multiple tools in sequence, and improvising or assembling a tool on the spot. A common recent approach uses a large language model for high-level reasoning, with imitation learning or reinforcement learning handling the specific motions. Note this is unrelated to “tool calling” in the large-model world, where an AI calls an API.

ExampleIn RoboTool (2023), a Kinova robot arm cannot reach a milk carton at the far end of a table, so it selects a hammer from among several objects and uses it to hook the carton within reach.

Related

Affordance · Manipulation · Contact-rich Manipulation · LLM-based Task Planning · Long-horizon Task · Dexterous Manipulation

Sources

Creative Robot Tool Use with Large Language Models (arXiv 2310.13065)