Audio-Visual Navigation

An agent using both 'eyes' and 'ears' together to find a sound-emitting target in a 3D environment.

This adds hearing on top of visual navigation: the agent receives both a first-person view and audio at the same time, and has to walk to an object that is making a sound. Sound can travel around walls and reflect off surfaces in a room, so it both hints at direction and reveals the space's structure, which is especially useful when the target is out of view. SoundSpaces (ECCV 2020) is an early landmark in this area: built on geometric acoustics, it renders sound for two sets of real scans, Matterport3D and Replica, and plugs into the Habitat simulator, where agents are trained with multimodal deep reinforcement learning. Later work on “semantic audio-visual navigation” has objects emit short, semantically appropriate sounds, such as a toilet flushing or a door creaking, and requires the agent to keep heading to the source from memory even after the sound has stopped.

ExampleIn SoundSpaces, an agent is placed in an unfamiliar apartment and hears a sound coming from somewhere; it has to combine vision and audio to walk all the way to the source and stop there.

Related

Navigation · Object-Goal Navigation · Point-Goal Navigation · Multimodal Perception · Microphone Array · Habitat

Sources

SoundSpaces: Audio-Visual Navigation in 3D Environments (arXiv 1912.11474)
Semantic Audio-Visual Navigation (arXiv 2012.11583)