06Perception & Sensors

A robot's sense of its own joint positions, velocities, forces, and posture, as opposed to sensing the outside world.

A robot’s sensing of its surrounding environment using sensors like cameras and lidar.

Understanding a scene and task using vision together with touch, force, sound, and other senses at once.

The field of making computers extract useful information from images and video and understand what a scene contains.

Using cameras and software on a factory floor to automatically inspect, measure, identify, and guide robots.

An ordinary camera that outputs color images, recording red, green, and blue brightness at every pixel.

A camera mounted on a robot arm's wrist or gripper that moves with the hand, giving a close-up view of the workspace.

A camera mounted off the robot’s body, fixed in place, viewing the whole workspace from the side.

A camera mounted on a robot's head or upper body that gives a near-first-person view of the scene.

A camera mounted in a robot’s palm that gives a close-up view of the hand and object during grasping.

A target being partly or fully blocked from a sensor's view by something else, including the robot's own body.

Observing the same scene from multiple cameras or multiple angles at once.

A mathematical model treating a camera as an ideal tiny hole, projecting 3D points onto an image plane along straight lines.

A 3D coordinate frame centered on the camera's optical center, with the z axis along the optical axis, used to locate objects relative to the camera.

The parameters describing how a camera itself forms an image, determining which pixel a 3D point lands on.

The rotation-plus-translation parameters describing where a camera sits in space and which way it points.

Projection maps a 3D point onto a pixel; back-projection uses a pixel plus its depth to recover the 3D point.

The way a real lens's image deviates from the ideal pinhole model, bending straight lines into curves.

The angular range a camera or sensor can see at once, usually given as horizontal, vertical, and diagonal figures.

A camera with an ultra-wide fisheye lens, often around 180° field of view, whose images bow visibly at the edges.

The chip inside a camera that turns light into a digital image, with amplification built into every pixel — the dominant sensor today.

The processing unit that turns an image sensor’s raw output into a normal color image.

The two ways a camera can expose an image: all pixels at once, or row by row in sequence.

A high-speed serial interface standard that carries image data from a camera sensor to a processor.

An automotive-grade serial link that powers a camera and carries high-speed image data over a single coaxial cable.

A camera where each pixel independently outputs an “event” only when brightness changes, instead of capturing full frames.

A camera that captures infrared heat radiation instead of visible light, revealing an object’s temperature distribution.

The umbrella term for techniques that recover an object's or scene's 3D geometry from images or sensor data.

An image whose pixel values store distance to the camera instead of color.

Data made of many 3D coordinate points used to represent the shape of an object or scene.

A camera that outputs, for every pixel, both a color and the distance from that point to the camera.

Two side-by-side cameras that shoot the same scene at once, computing distance from how much the two images differ.

The horizontal position difference of the same point between the left and right images — bigger for closer objects.

The distance between a stereo camera’s two lens centers, which sets how far and how accurately it can measure depth.

Finding the same point’s position in both the left and right images to compute disparity, then converting that into depth.

A stereo camera pair with an infrared projector adding a speckle pattern to help match the left and right images and compute depth.

A widely used stereo depth camera series, formerly an Intel division, common in robotics research and products.

A stereo depth camera series from Orbbec that works indoors and out, commonly mounted on robots.

A passive stereo depth camera line from the French company Stereolabs, common in mobile robots and data collection.

A stereo depth camera from Luxonis with a built-in AI chip, letting neural networks run directly on the camera.

Projects a known pattern onto a scene and computes depth from how the pattern gets distorted.

Projecting a random infrared speckle pattern onto a scene and computing depth from how it deforms or matches.

A laser projects a dot or line, a camera views the spot from the side, and triangulation gives distance and profile.

Measures how long light takes to bounce off an object and return, then converts that time into distance.

Emits a light pulse and directly times its echo, computing distance from the round-trip time.

A depth-sensing method that emits modulated light and computes distance from the phase shift of its echo.

A developer-oriented RGB-D camera from Microsoft that combines a depth sensor, color camera, microphone array, and IMU in one device.

An indirect time-of-flight RGB-D camera from Orbbec, positioned as a drop-in replacement for Microsoft’s discontinued Azure Kinect.

A semiconductor laser that emits light perpendicular to the chip surface, a common light source in depth cameras and lidar.

A photodetector sensitive enough to register a single photon, at the core of dToF lidar and ranging chips.

A robot vision sensor from RoboSense that combines active depth sensing, a color camera, and an IMU in one module.

Pixels where a depth camera can’t measure distance, usually recorded as zero in the depth map.

Erroneous depth points at object edges in a depth map, floating between the foreground and background.

A sensor that fires lasers and times their return to measure distance, scanning out a 3D point cloud of its surroundings.

A spinning laser rangefinder that scans one horizontal plane, measuring the distance to obstacles all around.

The number of laser channels arranged vertically in a multi-line lidar, which sets how dense the point cloud is top to bottom.

Lidar with no bulk rotating part, scanning instead with electronics or a small internal mechanism.

A lidar that continuously emits frequency-swept laser light, measuring both distance and velocity at every point.

A small 360° hybrid solid-state lidar from Livox with a built-in IMU, widely used on mobile robots.

A compact, ultra-wide-field-of-view 128-channel 3D lidar from Hesai Technology aimed at robots.

A hemispherical-field-of-view digital lidar from RoboSense for robots, covering 360°×90° with a single unit.

Unitree’s own low-cost, hemispherical-field-of-view lidar, where every point carries a 3D coordinate plus a grayscale value.

A radar that transmits millimeter-wavelength electromagnetic waves to measure a target’s distance, speed, and bearing.

A cheap, short-range sensor that measures distance by emitting ultrasound and listening for its echo.

A sensor that detects whether something is nearby, and how close, without needing to touch it.

Certified safety devices that detect a person entering a danger zone and stop the machine before contact.

Perceives the environment using only cameras, without lidar, millimeter-wave radar, or other ranging sensors.

A position sensor mounted on a motor or joint that converts shaft angle into an electrical signal.

A sensor combining an accelerometer and gyroscope that measures an object's acceleration and rotation rate.

Fuses gyroscope, accelerometer, and (optionally) magnetometer data to compute roll, pitch, and yaw angles in real time.

A simple filter that blends the gyroscope for short-term accuracy and the accelerometer for long-term stability to estimate attitude angle.

A sensor’s reading slowly shifting over time or temperature even when the true input hasn’t changed.

A statistical method for how sensor noise changes with averaging time, commonly used to calibrate IMU noise parameters.

A sensor that measures force along three axes and torque around three axes at once, often mounted on an arm's wrist.

A sensor built into a robot joint that directly measures the torque the joint is actually producing.

A small element bonded to metal whose electrical resistance changes as it deforms, the basic building block of force sensors.

When a multi-axis force sensor loaded along only one direction still shows nonzero readings on the other axes.

Estimating the external force on a robot from motor current and a dynamics model, without installing a dedicated force sensor.

Determining whether, when, and where a robot has touched an object or its environment.

A sensor mounted on a legged robot’s foot or ankle that measures the contact force between foot and ground.

Determines whether each foot of a legged robot is currently on the ground or in the air.

A sensor that lets a robot feel contact, measuring pressure distribution, contact location, shear force, and slip.

A soft, bendable, large-area tactile sensing layer applied to a robot's surface, mimicking human skin.

Many small tactile sensing elements arranged in a grid, measuring the pressure distribution across a contact surface.

The smallest individual sensing element in a tactile array — the tactile equivalent of an image pixel.

Measuring pressure from a material’s change in electrical resistance under load; cheap and easy to build into large sensor arrays.

A tactile-sensing method that infers contact force from the change in capacitance as pressure alters electrode spacing or a dielectric layer.

Sensing contact and vibration using the piezoelectric effect, where a material generates an electric charge when it is deformed by force.

Mixes magnetic particles into a soft gel and uses a magnetometer to sense contact from the magnetic-field change deformation causes.

A magnetic tactile skin: a soft pad embedded with magnetic particles sits over a magnetometer that reads contact from field changes.

A magnetic tactile skin from XELA Robotics where every sensing point measures both pressure and shear force at once.

A multi-dimensional tactile sensor from PaXini that outputs distributed force, net force, and torque at the point of contact.

SynTouch’s biomimetic fingertip tactile sensor, which senses contact force, micro-vibration, and temperature.

A tactile sensor that uses a built-in camera to photograph how a soft surface deforms on contact.

Keeping the camera fixed and lighting a surface from several directions, then recovering its shape from how the shading changes.

A vision-based tactile sensor that reads touch from a camera watching an elastic gel deform, also the name of the company that makes it.

GelSight’s commercial compact vision-based tactile sensor, ready to use over USB.

Meta's open-source, fingertip-sized, low-cost vision-based tactile sensor that reads contact from a camera watching a gel deform.

Meta’s humanoid fingertip-shaped multimodal tactile sensor, sensing force, vibration, temperature, and more.

A slim, finger-shaped vision-based tactile sensor from MIT that can be mounted on a gripper for grasping.

An open-source, compact vision-based tactile sensor from Tsinghua’s Huazhe Xu group that measures contact shape and 6D force.

A line of vision-based tactile sensors from Daimon Robotics that senses force by imaging how a contact surface deforms.

A biomimetic optical tactile fingertip from the Bristol Robotics Laboratory that uses a camera to watch internal marker pins move.

Prints a dot pattern on a vision-based tactile sensor’s soft gel and tracks the dots’ displacement to estimate shear force and slip.

Determining whether an object held in a hand is starting to slide, so the grip can be adjusted in time.

The perception task of inferring where and how an object held in the hand is making contact with the outside environment.

Arranging tactile sensor readings into a 2D image, so ordinary vision models can process them directly.

A general-purpose vision-based-touch representation model from Meta, pretrained with self-supervision on a large set of tactile images.

A unified tactile-representation model spanning multiple vision-based tactile sensors, from Renmin University’s Di Hu group, covering both static and dynamic touch.

A microphone stuck to a gripper or object that picks up contact vibration, used as a cheap tactile sensor.

Combining what a camera sees with what a tactile sensor feels into a single, unified signal.

Estimating a camera's intrinsics, distortion coefficients, and extrinsics so pixels and 3D coordinates can convert into each other.

A flat board printed with a pattern of precisely known size, used to give a camera calibration accurate reference points.

The pixel distance between a 3D point reprojected onto the image and where it was actually observed.

A printable black-and-white square marker that a camera can detect to compute the marker's ID and 6D pose.

A square tag with a black border around a black-and-white grid code; a camera that spots it can compute its own pose relative to the tag.

Recovering a camera’s pose from a set of known 3D points and where each one lands in the image.

Finding the fixed coordinate transform that relates a camera to a robot arm.

A camera-mounting scheme where the camera is fixed to the robot arm's end-effector and moves along with it.

A camera-mounting scheme where the camera is fixed outside the robot and doesn't move with the arm, plus its matching calibration.

Reprojects a depth map into the color camera’s viewpoint so the two images’ pixels line up one to one.

Finds the relative pose and time offset between a camera and an IMU so their data can be aligned and fused.

Finds the rotation and translation between a lidar and a camera so a point cloud can be projected accurately onto the image.

Making sure data from different sensors used together actually comes from the same moment in time, not slightly different ones.

Finding what objects appear in an image and where, marking each with a bounding box.

The rectangle an object detector draws around an object, given as a few coordinates marking its location in the image.

A family of real-time object detectors that locate every object in an image with a single pass through the network.

The overlap area of two regions divided by their combined area, used to judge how accurate a predicted box or mask is.

A post-processing step that keeps only the highest-scoring box among a cluster of overlapping detection boxes.

An end-to-end detector that frames object detection directly as “set prediction” using a Transformer.

Precision measures how many reported results were correct; recall measures how many of the true targets were found.

The most common accuracy metric for object detection and segmentation: average precision (AP) per class, averaged across classes.

Labels every pixel in an image with a category, like “table,” “cup,” or “floor.”

Cutting out every object in an image pixel by pixel, keeping separate objects of the same class distinct from each other.

A segmentation task that labels every pixel with a category while also telling individual object instances apart.

A pixel-by-pixel map, the same size as the image, marking which pixels belong to a given object.

An instance segmentation model from 2017, proposed by Kaiming He and colleagues, that outputs both detection boxes and a per-object mask.

Meta's 2023 promptable segmentation foundation model: give it a point or box and it returns that object's mask.

Locating a handful of pre-defined, meaningful points in an image, like a wrist, a cup's handle, or a box's corner.

Continuously finding the same object across video frames and keeping its identity consistent over time.

Continuously cutting out a pixel mask for a specified object in every frame of a video.

Meta's image-and-video segmentation model: mark a target once and it keeps segmenting and tracking it through the whole video.

How far, and in which direction, every pixel appears to move between two consecutive video frames.

A classic optical-flow network that computes correlation between every pair of pixels, then refines the flow through repeated recurrent updates.

The 3D motion vector of every point in a scene between two adjacent frames.

Given any point in a video, outputting its position in every later frame, and whether it becomes occluded.

A DeepMind point-tracking model that first finds a rough match per frame, then refines the trajectory over time.

Meta’s open-source video point-tracking model that jointly tracks large numbers of pixels, including ones that get occluded.

Continuously tracks arbitrary pixels through a video and outputs their motion trajectory in 3D space.

Detecting whatever object a piece of text names, instead of being limited to a fixed list of trained classes.

An open-vocabulary detector that finds and boxes whatever object a text description names, given an image and text.

A Google open-vocabulary object detection model that can find objects from a text description or an example image.

An open-vocabulary detector that detects objects in real time just from a typed category name.

IDEA Research’s open-world detection model that can box objects from text, examples, or no prompt at all.

Segmenting exactly the pixels that match any text description of a category, even one the model never saw during training.

An open-source pipeline that boxes and finely segments the object a text phrase refers to in an image.

Meta's third-generation segmentation model that finds every instance matching a short phrase or an example image.

Finds the location in an image that a phrase or sentence refers to.

Given a sentence describing one specific object, precisely segmenting out exactly that object in the image.

Inferring how far every pixel in an image is from the camera, producing a depth map.

Predicting how far away every pixel is from just one ordinary color photo.

Metric depth gives true distance in meters; relative depth only gives near/far ordering, off by an unknown scale and offset.

A monocular depth estimation foundation model from HKU and TikTok that produces a depth map from a single photo.

Apple’s open-source monocular metric-depth model that outputs a sharp depth map in meters from a single image.

A model that estimates metric depth from a single image, resolving scale ambiguity across cameras with a canonical camera-space transform.

A Microsoft Research model that estimates full 3D geometry from a single image, predicting a 3D point for every pixel.

NVIDIA’s stereo-matching foundation model that produces depth in a new scene with no fine-tuning needed.

Using a cheap lidar’s sparse depth readings as a “prompt” so a large depth model outputs accurate 4K metric depth.

Fills in missing or sparse pixels in a depth map to produce complete, dense depth.

An open-source depth-completion model from Ant Group’s Robbyant that uses a color image to repair a depth camera’s holes and noise.

A depth-repair model from ByteDance Seed that cleans up noisy depth-camera output into near-simulation-quality accurate depth.

Getting a robot to correctly see glass, metal, and other objects that ordinary depth cameras get wrong.

Depth and segmentation datasets collected specifically for glass, metal, and other objects that depth cameras struggle to measure.

Corners, blobs, and other points in an image that can be reliably found again, together with a vector describing their surroundings.

Finds pairs of feature points across two images that correspond to the same physical point.

A line of neural-network models for detecting and matching feature points across images, replacing hand-crafted methods like SIFT.

Finds, for every pixel in one image, the matching location in another image.

Repeatedly fitting a model to small random subsets of data and keeping the one most data points agree with, to reject outliers.

The geometric relationship that forces the matching point of a scene point, seen by two cameras, to lie on a specific line.

A 3×3 matrix describing how pixels on the same plane correspond between two images.

Recovering a point’s 3D coordinates from two cameras’ known positions and where that point lands in each image.

Jointly fine-tunes all camera poses and 3D point coordinates to minimize reprojection error.

Computing both camera poses and a scene’s 3D points at once from a set of photos taken from different angles.

A method that recovers a scene’s dense 3D structure from multiple photos whose camera positions are already known.

A 3D reconstruction approach that outputs camera parameters, depth, and a point cloud directly from images in a single network forward pass.

An array the same size as an image, where each pixel stores a 3D coordinate instead of a color.

A feed-forward reconstruction model that regresses a per-pixel 3D point map directly from two images, with no camera parameters needed.

Naver’s model that adds a matching head onto DUSt3R, outputting both a 3D point map and dense matching features.

A 3D vision model that computes camera parameters, depth, and a point cloud from multiple images in a single forward pass.

A feed-forward 3D reconstruction model with no reference viewpoint, giving the same result no matter what order the images come in.

A 3D reconstruction model with a persistent memory state that reads images one at a time and updates the whole scene online.

Meta and CMU’s general-purpose feed-forward 3D reconstruction model that directly outputs true-scale scene structure.

ByteDance Seed’s geometry model that recovers depth and camera poses from any number of images, with poses optional.

A small cube in 3D space — the volumetric equivalent of a pixel.

Preprocessing steps that crop irrelevant regions, remove noisy points, and shrink a raw point cloud down to a usable size.

Repeatedly picks the point farthest from the already-selected set, downsampling a point cloud evenly to a fixed number of points.

Computing which way each point on an object’s surface faces — the vector perpendicular to the surface there.

A hand-crafted feature that describes each point’s local shape using a histogram of normal-vector geometric relationships in its neighborhood.

Finding the rotation and translation that lines up two point clouds within one shared coordinate frame.

A classic registration algorithm that repeatedly finds nearest-point pairs and solves for rotation and translation to align two point clouds.

A point-cloud registration method that divides space into a grid, models each cell as a normal distribution, and aligns scans against it.

Finds each point’s nearest neighbor in the other point cloud and averages the distances, to measure how close two shapes are.

A pioneering network that classifies and segments point clouds directly, with PointNet++ its hierarchical upgrade.

A transformer backbone for point clouds that swaps costly neighbor search for a fast serialization trick, letting it see farther and run faster.

Labeling every point in a point cloud with which object category, or which individual object, it belongs to.

Finds objects in 3D space and outputs their category along with an oriented 3D box.

Given a partial point cloud, inferring and filling in the parts of an object that were occluded or never scanned.

Represents a 3D object’s surface using vertices connected into triangular faces.

Storing the signed distance to the nearest surface in a voxel grid, used to fuse multiple depth frames into a 3D model.

Whether a 3D scene is stored directly as geometric elements, or as a function you query by coordinate.

A neural network that memorizes the color and density of every point in a scene, letting it render any new viewpoint.

Representing a scene as many colored 3D Gaussian ellipsoids that can be rendered from any new viewpoint in real time.

Generating an image of a scene from a camera viewpoint that was never actually photographed, using a handful of existing photos.

Reconstructs a 3D scene that moves — the geometry plus how it changes over time.

Given just one photo, having a model fill in an object’s complete 3D shape and texture.

Tencent’s open-source image-to-3D asset model series, generating a textured 3D mesh from a picture.

Meta’s open single-image 3D reconstruction models, released as separate versions for objects and for human bodies.

Computing an object's 3D position and 3D orientation relative to the camera — six degrees of freedom in total.

NVIDIA's general-purpose 6D pose model that estimates and tracks the pose of new objects without retraining.

A method that estimates 6D pose for a new object given only its CAD model, with no retraining needed.

A method that uses SAM’s segmentation to estimate the 6D pose of objects it was never trained on, given only a CAD model.

Continuously estimating an object’s 3D position and orientation across a video’s successive frames.

Estimates 6D pose and size for an unseen object in a known category, without needing that exact object’s CAD model.

A shared, standardized coordinate frame for all objects in a category, used to estimate the pose and size of objects never seen before.

Poses an object’s 3D model under both the true and predicted pose and averages the point-to-point distance, to score 6D pose estimation.

A public benchmark and yearly challenge for 6D object pose estimation, maintained by the Czech Technical University.

Uses a 3D camera to find a workpiece’s position and orientation and guides an industrial robot to pick or process it.

Computing directly from an image or point cloud where and in what orientation a gripper should grasp an object.

A general-purpose grasp-detection model from Shanghai Jiao Tong University that outputs many usable grasp poses directly from a point cloud.

NVIDIA’s grasping network that generates 6-DoF grasps directly from a depth point cloud in cluttered scenes.

NVIDIA’s framework that generates 6-DoF grasp poses with a diffusion model, then scores and filters them.

Representing an object with a handful of meaningful points on it — a mug’s handle, a kettle’s spout — to make planning manipulation easier.

Finds where on an object you can grip, press, or pour from an image or point cloud, and marks it to a specific region.

Infers an object’s parts, how they’re joined, which axis each part moves around, and how far it’s currently open.

Finding the positions of a person's body joints from an image or video and connecting them into a skeleton.

Estimating the positions of a human hand's joints and how the fingers are bent, from an image or sensor data.

Google's open-source on-device toolkit that extracts hand and body keypoints in real time from an ordinary camera feed.

Recovering 3D human motion straight from ordinary video, with no reflective markers worn on the body.

A parametric 3D human body model that generates a full body mesh from a small set of shape and pose parameters.

The task of estimating a person’s complete 3D body mesh — pose plus body shape — from an image or video.

A method that recovers a person’s 3D motion in world coordinates from monocular video.

A Transformer model that reconstructs a 3D hand mesh from a single RGB image.

Quickly finding every hand in an image and reconstructing each one as a 3D hand mesh.

The study of how human hands contact, grasp, and manipulate objects, including estimating their joint 3D pose and contact.

Lets a machine recognize a gesture a person makes from images or sensor signals and understand its meaning.

Measures the movement of a person’s eyes and estimates where they are looking right now.

Multiple microphones arranged in a fixed geometry working together to pick up sound directionally and judge where it comes from.

Automatically converts spoken words into text — the first step in a robot understanding a spoken command.

Estimates a robot’s current position, orientation, and velocity from noisy sensor readings.

Combining data from a camera, lidar, IMU, and other sensors to get an estimate more accurate and stable than any one alone.

Counts how much the wheels have turned to estimate how far a robot has traveled and how much it has turned.

Uses joint encoders, the IMU, and contact detection to work out how far and in what direction a legged robot has moved.

A neural network that estimates quantities like body velocity from joint and IMU data, often trained jointly with a locomotion policy.

Compares consecutive camera frames to estimate, frame by frame, how far and how much a camera has turned.

A robot building a map of an unfamiliar environment while simultaneously figuring out where it is on that map.

The two-part division of labor in a SLAM system: the front end estimates motion from sensor data, the back end globally corrects error.

Lets a robot recognize that it has returned to a place it has already been, used to remove SLAM’s accumulated drift.

Looking at an image and determining which place it is, and whether the system has been there before.

Figuring out where a robot is and which way it’s facing on an existing map, after tracking is lost or it restarts.

Uses only cameras to simultaneously estimate where it is and build a map of the surroundings.

A classic open-source visual SLAM system from the University of Zaragoza, supporting monocular, stereo, RGB-D, and IMU input.

Using a lidar's scanned point clouds to localize a robot and build a map of the environment at the same time.

A classic method that splits lidar SLAM into a high-frequency odometry step and a low-frequency mapping step, plus its lightweight variants.

A map that divides the environment into cells, each storing the probability that it's blocked by an obstacle.

A recursive estimation method that approximates a probability distribution over states using a large set of weighted random samples.

A ROS localization module that estimates a robot's position on a known map using particle filtering and laser scans.

Combines satellite navigation with base-station differential correction to achieve centimeter-level positioning outdoors.

Using extremely short radio pulses to measure signal flight time, for centimeter-level indoor ranging and positioning.

Two architectures for multi-sensor fusion: combining raw measurements together, versus computing each sensor’s own result and merging afterward.

A Kalman filter formulation that estimates the difference between a nominal state and the true state, rather than the state itself.

A Kalman filter for nonlinear systems that propagates a small set of sample points instead of taking derivatives.

An extended Kalman filter that defines error on a Lie group, converging more reliably, commonly used for legged-robot state estimation.

Draws every measurement constraint as a “variable-factor” graph, then finds the most likely state by least squares.

A technique that pre-integrates a large batch of IMU readings between two frames into a single relative-motion constraint.

Fusing a camera and an IMU to continuously estimate a device’s own motion trajectory.

An open-source visual-inertial localization system from HKUST that fuses a camera and an IMU to estimate pose in real time.

Fuses lidar point clouds with IMU readings to estimate a robot’s pose in real time while building a point-cloud map.

An open-source lidar-plus-IMU odometry system from HKU that's fast and works with many lidar types.

An open-source tightly coupled lidar-inertial odometry and mapping system built on factor-graph optimization.

An open-source graph-based SLAM library that supports mapping and localization with RGB-D, stereo, or lidar sensors.

A deep-learning visual SLAM system from Princeton that iteratively estimates camera pose and dense depth.

A real-time monocular dense SLAM system built on MASt3R as a prior, able to map and localize from ordinary uncalibrated video.

A family of SLAM methods that use 3D Gaussian splatting as the map, building a photorealistically renderable scene while localizing.

Two standard metrics for how far a SLAM or odometry system’s estimated trajectory deviates from ground truth.

Working out what's in an environment, where it is, and how things relate to each other from images, depth, or point clouds.

A 3D probabilistic occupancy map stored hierarchically as an octree, commonly used for robot obstacle avoidance and planning.

An open-source, GPU-accelerated 3D mapping library from NVIDIA that builds real-time distance-field maps for obstacle avoidance.

A neural network that predicts, for every location in 3D space, whether it is occupied by an object.

A 2D top-down grid that fuses information from multiple sensors onto a single ground plane.

A 2.5D terrain map that divides the ground into a grid and stores one height value per cell, commonly used by legged robots.

Samples ground height around a robot in a fixed pattern, used as terrain input for a locomotion policy.

Judging which parts of the terrain ahead can be crossed, which can’t, and how costly each path would be.

A map made of place nodes and connecting edges, recording only what leads where, not exact coordinates.

A map that labels what each place is, on top of its geometry, so it can be queried by object name or natural language.

SLAM that recognizes object categories while localizing and mapping, producing a map with semantic labels attached.

Organizes a 3D scene’s floors, rooms, objects, and their relationships into a layered graph.

Fuses multi-view images into an open-vocabulary 3D object scene graph for large-model reasoning and robot planning.

Distills features from 2D models like CLIP and DINO into a 3D scene, so every point in space carries semantics.

Embeds CLIP language features into a NeRF, so objects in a 3D scene can be found with natural language.

Finds the object in a 3D scene that a sentence describes.

Given an image and a natural-language question about it, the model answers in words.

A vision-language model answering by marking exact pixel coordinates on the image, following a text instruction.

Drawing boxes or numbers directly on an image so a multimodal large model can answer by referring to those marks.

A benchmark that uses indoor videos to test how well multimodal large models understand space.

A robot deliberately moves its sensors or body to see or feel a target better, rather than just passively observing.

Deciding where a sensor should look next, given what it has already seen, to gain the most useful information.

A robot actively pushes, pokes, or pulls objects, using the resulting changes to understand its environment.