Optical Flow

How far, and in which direction, every pixel appears to move between two consecutive video frames.

Optical flow describes the apparent motion of the brightness pattern in an image when the camera and the scene move relative to each other, and the result is usually a 2D displacement field the same size as the image. Classic methods rely on a ‘brightness constancy’ assumption — the same point keeps the same brightness across adjacent frames — but one equation can't solve for two unknowns (the aperture problem), so the 1981 Lucas-Kanade method assumes a small local window moves consistently, while Horn-Schunck instead adds a global smoothness constraint. The deep-learning-era representative is RAFT (ECCV 2020), which estimates dense optical flow using correlation volumes between all pairs of pixels plus recurrent iteration. Flow computed at only a few feature points is called sparse; flow computed at every pixel is called dense. Robots use optical flow for visual odometry, obstacle avoidance, and motion segmentation, and it's also used as an intermediate action representation: some manipulation policies first predict how points on an object will move, then convert that prediction into a robot action.

ExampleAs a drone flies forward, objects ahead appear to expand outward in the frame, with closer objects showing larger optical flow — this can be used to judge whether a collision is imminent.

Also called

Dense Optical Flow, Sparse Optical Flow

Related

RAFT · Scene Flow · Tracking Any Point · Visual Odometry · Event Camera · Feature Matching

Sources

Wikipedia: Optical flow
RAFT: Recurrent All-Pairs Field Transforms for Optical Flow