Eye Tracking / Gaze Estimation

Measures the movement of a person’s eyes and estimates where they are looking right now.

Eye tracking measures the eyeball’s movement relative to the head, while gaze estimation goes a step further and computes where the line of sight lands — where the person is actually looking. The dominant method shines infrared light on the eye and uses a camera to capture both the pupil center and the corneal reflection at the same time, computing gaze direction from their relative position; another approach uses a neural network to regress gaze directly from an ordinary image. In embodied AI it has two main uses: recording a demonstrator’s gaze point while collecting demonstration data, as an extra signal for “where in the frame to pay attention,” helping a policy concentrate computation on the key region; and judging a person’s intent and the object of their attention during human-robot interaction. Head-worn devices like Meta’s Project Aria glasses have a built-in eye-tracking camera, so gaze information can be recorded alongside first-person data.

ExampleKim and colleagues (RA-L 2021) recorded an operator’s gaze point during teleoperation; the resulting trained policy used high-resolution image detail near the gaze point and lower resolution at the periphery, and completed a high-precision task like robotic needle threading.

Also called

Gaze Tracking

Related

Egocentric Video · Project Aria Glasses · Human-Robot Interaction · Intent Understanding · Imitation Learning · Active Perception

Sources

Wikipedia: Eye tracking
Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation (RA-L 2021)
Project Aria Gen 1 hardware specifications