MediaPipe

Google's open-source on-device toolkit that extracts hand and body keypoints in real time from an ordinary camera feed.

MediaPipe is Google's open-source (Apache 2.0), cross-platform, on-device machine learning framework, and MediaPipe Tasks provides ready-made vision tasks within it. Two are most relevant to embodied AI: Hand Landmarker first detects the palm, then regresses 21 hand keypoints within the cropped region, outputting image coordinates, metric world coordinates in meters, and left/right handedness; Pose Landmarker, built on BlazePose, outputs 33 body keypoints. Both run in real time on a phone's CPU and support mobile, web, and Python. Because it needs nothing more than an ordinary RGB webcam, MediaPipe is commonly used for low-cost gesture-based teleoperation, retargeting human hand motion onto a dexterous hand, or labeling keypoints on human videos; its 3D accuracy is limited under occlusion and complex gestures, so 3D hand-reconstruction models like HaMeR are used instead when higher accuracy is needed. The older Solutions interface stopped being supported in March 2023.

ExampleAn example program from dex-retargeting uses MediaPipe to detect human hand keypoints in real time from a laptop webcam, then retargets them into joint angles for an Allegro dexterous hand, making the simulated robotic hand mimic the person's hand movements.

Also called

Google MediaPipe, MediaPipe Hands, MediaPipe Pose, Hand Landmarker, Pose Landmarker, BlazePose

Related

Hand Pose Estimation · Human Pose Estimation · Keypoint Detection · Motion Retargeting · HaMeR · dex-retargeting

Sources

MediaPipe Hand Landmarker guide
MediaPipe Pose Landmarker guide
google-ai-edge/mediapipe (GitHub)

As of

2026-09