Forward Dynamics Model

A model that takes the current state and an action and predicts what the next state will look like.

A forward dynamics model learns ‘what the world will look like after this action is taken’: it takes the current state (or observation) s_t and an action a_t as input and outputs the next state s_{t+1}. The idea parallels forward dynamics in mechanics, where torques are used to compute acceleration, but in robot learning it usually isn't derived from physics equations — instead, a neural network learns it from interaction data. With such a model, a robot can try out many candidate action sequences inside the model first and execute whichever leads to the best predicted outcome; this is exactly what model-based reinforcement learning and model predictive control (MPC) do. In 2017, Nagabandi et al. combined a learned neural-network dynamics model with MPC to make simulated legged robots track arbitrary trajectories. Today's world models are essentially large forward dynamics models operating on images or a latent space; a forward dynamics model is often paired with an inverse dynamics model, and the decoder of a latent action model belongs to this family too.

ExampleIn a block-pushing task, the model is given the block's current position and the action ‘push right 2 cm’ and predicts the block's new position afterward; MPC uses it to compare dozens of candidate pushes and execute whichever gets closest to the goal.

Also called

FDM, Forward Model, Learned Dynamics Model

Related

Inverse Dynamics Model · World Model · Model-Based Reinforcement Learning · Model Predictive Control · Forward Dynamics · Latent Action Model

Sources

Neural Network Dynamics for Model-Based Deep RL with Model-Free Fine-Tuning (arXiv 1708.02596)
Curiosity-driven Exploration by Self-supervised Prediction (arXiv 1705.05363)