Termination vs. Truncation

An episode can end two ways: termination means the task itself is over; truncation means it was cut off by a time limit.

This is a pair of concepts from the reinforcement-learning environment interface. Termination means the agent has entered a terminal state of the Markov decision process — the task is complete, or the robot has fallen over. Truncation means the episode was cut off for a reason outside the task itself, most commonly a maximum step count set for training. The two affect training differently: after termination there is no future reward, so the value target is just the immediate reward; a truncated state is not actually an endpoint, so its target should still bootstrap using the value estimate of the next state (substituting an estimate for return that hasn't actually been observed yet). Pardo and colleagues' ICML 2018 paper specifically analyzed the problems caused by treating a time limit as termination. Starting with version 0.26, Gymnasium's step() returns terminated and truncated separately, replacing the old single done flag; Isaac Lab similarly uses a time_out flag to distinguish the two.

ExampleIn legged_gym, an episode terminates when contact force on designated body parts exceeds a threshold; it truncates when the step count exceeds the limit, which is passed to rsl_rl through a time_outs field in extras — PPO adds the discounted value estimate back into the reward at truncated steps, effectively continuing to bootstrap.

Also called

terminated / truncated, Time-limit Truncation

Related

Episode · Early Termination · Value Function · Bootstrapping (in Reinforcement Learning) · Environment (Env; reset/step interface) · Gymnasium

Sources

Gymnasium: Handling Time Limits
Time Limits in Reinforcement Learning (arXiv 1712.00378, ICML 2018)
rsl_rl PPO 实现（time_outs 自举） (Chinese)