Intrinsic Motivation

A reward the agent generates for itself from novel or hard-to-predict states, used to drive exploration.

Intrinsic reward is a reward signal computed internally by the agent rather than coming from the task itself, usually measuring how novel a state is or how poorly the agent can predict it; it is typically added on top of the environment's extrinsic reward. It mainly addresses exploration under sparse rewards, when task reward shows up so rarely that random trial and error almost never stumbles onto it. Representative work includes Pathak and colleagues' 2017 ICM, which treats the error in predicting the consequences of one's own actions as curiosity, and OpenAI's 2018 RND, which uses the error in predicting a fixed random network's output as the reward — becoming the first method to beat average human performance on the Atari game Montezuma's Revenge without demonstrations. In robotics it is commonly used for exploration and for unsupervised skill discovery.

ExampleIn Super Mario Bros., ICM gives the agent no game score at all, only a reward for failing to predict the next frame's features, and the agent still actively explores deeper into the level.

Also called

Curiosity-Driven Exploration, Intrinsic Reward

Related

Exploration vs. Exploitation · Sparse Reward · Unsupervised Skill Discovery · Reward Shaping · Reinforcement Learning · Active Exploration

Sources

Curiosity-driven Exploration by Self-supervised Prediction (ICM, arXiv:1705.05363)
Exploration by Random Network Distillation (RND, arXiv:1810.12894)