Decision Diffuser

Generates future state trajectories with a return-conditioned diffusion model, then infers actions from them, skipping dynamic programming.

Decision Diffuser was proposed in November 2022 by Pulkit Agrawal's group (Improbable AI Lab) and CSAIL at MIT, an oral presentation at ICLR 2023. Offline reinforcement learning usually needs to learn a value function and do dynamic programming (repeatedly using the Bellman equation to estimate long-term return), which tends to be unstable to train. Decision Diffuser instead treats decision-making as conditional generation: a diffusion model generates only a sequence of future states, conditioned on a desired return, a constraint, or a skill, strengthened with classifier-free guidance; an inverse-dynamics model then infers the action to take from each pair of adjacent states. It beat the leading methods of the time on the D4RL offline reinforcement-learning benchmark, and even though it only ever saw a single constraint or skill at training time, at test time it could combine multiple conditions together. It's a representative follow-up to Diffuser in using diffusion models for decision-making, in the same lineage as later “generate the future, then infer actions with inverse dynamics” approaches like UniPi.

ExampleIn a Kuka-arm block-stacking experiment, each training trajectory satisfies only a single constraint of the form “A is on top of B,” but at test time several constraints are given as conditions together, and Decision Diffuser generates a stacking plan that satisfies all of them at once.

Also called

Is Conditional Generative Modeling all you need for Decision-Making?

Related

Diffuser · Diffusion Model · Classifier-Free Guidance · Inverse Dynamics Model · Offline Reinforcement Learning · Return Conditioning

Sources

Is Conditional Generative Modeling all you need for Decision-Making? (arXiv:2211.15657)
Decision Diffuser 项目主页（ICLR 2023 Oral） (Chinese)

As of

2023-07