Advantage Guidance: matching quality versus improving it

The question: can supervised policy training still improve behavior?

Advantage Guidance (AG), by Zhiyuan Zhou and Sergey Levine, concerns policy extraction in offline reinforcement learning: turn a learned assessment of action quality into a policy that can improve on the dataset. The attractive part of behavior cloning is its simple supervised objective. Its weakness is that reproducing the behavior distribution also reproduces bad choices. Actor–critic methods explicitly prefer actions with high estimated value, but optimizing a multi-step generative actor through a critic introduces its own approximation and optimization difficulties.

AG keeps a supervised flow-matching objective for the actor, attaches an advantage label to each action, and changes how the trained actor is sampled. The central distinction is between learning to respond to a quality request and using that response to favor better actions. These are different problems, requiring different interventions. This note follows v2, Sections 3–7 (opens in a new tab); the companion technical note derives the identities rather than treating them as guarantees about an implemented sampler.

Think of the reference policy as a distribution over plausible actions, not one recommended action. We would like to reallocate probability within that distribution toward better choices. Conditioning on a label tells a model which part of a distribution to describe. It does not automatically specify the probability reallocation that an optimization objective demands.

1. Two failure modes of advantage conditioning

For a fixed state, advantage measures relative action quality:

A conditional actor learns $\pi(a\mid s,y)$ from examples whose label $y$ depends on estimated advantage. At inference we might request a large $y$. That appealing procedure can fail twice.

Failure mode one: the actor ignores the condition. State and noise already explain much of the flow-matching target. A model can reduce its average training error without learning a useful dependence on advantage. Low supervised loss therefore does not certify useful control by the condition. The paper calls the mismatch between requested and achieved advantage the advantage alignment problem (Section 5).

A useful diagnostic is to hold the state and initial sampling noise fixed, vary only the requested advantage, and measure how much the resulting action and its evaluated advantage change. Compare that effect with changing the noise. Otherwise, ordinary sample variation can be mistaken for successful conditioning. Alignment still uses an estimated critic: a well-aligned actor can be aligned to the wrong notion of quality.

Failure mode two: matching is not maximizing. Even an ideal conditional model can prefer an action of exactly the requested quality over an action that is better. To see why, assume Gaussian label noise:

Bayes’ rule gives, at fixed state,

The squared error penalizes overshooting as well as undershooting. Asking for a still larger value does not remove this conceptual mismatch and may put the request outside the training distribution. The first failure is about whether the model learns the condition; the second survives even if it learns perfectly. These are the issues separated in Sections 4–5 and in the supplied example.

2. What distribution would policy improvement want?

Fix the state and an estimated advantage function. A behavior-constrained improvement objective is

Its normalized solution is

This is Eq. 1 of the paper. The reference supplies plausibility; the exponential factor favors quality. Smaller $\beta$ concentrates more strongly on estimated high advantage. It does not establish that those estimates are correct or that the resulting sequential policy is globally optimal. A baseline depending only on state cancels from the normalized distribution, so replacing advantage by $Q$ gives the same fixed-state tilt if all other assumptions are held fixed.

For intuition, with equally likely actions of utilities zero and one and $\beta=1$, the improved probabilities are approximately 0.269 and 0.731. The target is a distributional tradeoff, not an instruction to find an action with utility exactly one. The technical note derives this optimum and explains the support and normalizability conditions.

3. Contrastive guidance: cancel the matching penalty

Ordinary classifier-free guidance combines the unconditional score $g$ and one conditional score $g_y$. Under the Gaussian-label model, its clean-action score corresponds to

Increasing $w$ sharpens a matching preference. It does not change the quadratic preference into the linear-in-advantage exponent required above (Section 4, Eq. 3).

AG instead compares two conditions, $y_+>y_-$. Subtracting their log likelihoods cancels the shared $A^2$ term:

The contrastive combination

therefore has the desired clean-action score when

Section 6.1, Eq. 4 and Appendix A.1 state this more generally for an exponential-family likelihood. The mathematical contribution is the cancellation, not simply “ask for a higher reward.” Importantly, an exact identity between clean-density scores is not a proof that a learned, finitely integrated velocity field samples that density exactly. Keep the target, the learned model, and the numerical solver distinct.

4. Making the condition learnable: residual flow and imagined actions

For the straight interpolation $x_t=(1-t)\epsilon+ta$, with Gaussian $\epsilon$, the conditional flow-matching target is $a-\epsilon$. AG separates the conditional velocity into

The reference actor learns ordinary behavior cloning. The residual actor learns the additional dependence on advantage through a loss of the form

This decomposition reduces the need for the conditional component to rediscover unconditional behavior (Sections 6.2, Eqs. 5–6). It is an architectural and training bias, not an algebraic guarantee that conditioning cannot be ignored. “Residual” also does not mean an action-space correction is simply added after generating an action: the correction operates along the generative flow.

Action imagination supplies a different missing ingredient: alternatives at the same state. For each dataset pair, keep its action $a_0$, sample $K$ more actions from the reference policy, evaluate their estimated advantages, and use these labeled alternatives to train the residual actor (Section 6.3). These are counterfactual actions, not rollouts of imagined environment trajectories. They improve supervision only to the extent that the reference proposals are useful and the critic can evaluate them reliably.

Per-state standardization then gives

This puts the label on a comparable scale across sampled states. It also changes the units of the exponential tilt: a fixed coefficient on standardized advantage is not a fixed coefficient on raw return. The denominator is estimated from a finite set of actions and needs numerical stabilization.

5. Training and inference, in operational order

A useful implementation-level reading of Algorithm 1 is:

1. Learn value estimates from the offline data; the paper uses IQL for $Q$ and $V$. AG is a policy-extraction method, not a replacement for value estimation.
2. Sample dataset states and actions. Update the reference flow with its ordinary flow-matching objective.
3. Draw additional actions for each state from that reference. Evaluate $Q-V$ on the dataset action and sampled alternatives.
4. Standardize the advantages within each state’s action set and draw noisy labels. Train the residual conditional flow against the combined-velocity target above.
5. For a new state, draw initial Gaussian noise. At every integration step compute

6. Apply the numerical flow update and return the generated action or action chunk.

The two conditional evaluations use the same current noisy action, state and flow time. The unconditional term cancels in the conditional difference, but it remains once as the reference term. One should not accidentally count it twice. Nor should “no critic gradient through the sampler” become “critic-free”: the labels and the learned preference still depend on the critic.

The authors’ implementation (opens in a new tab) identifies agents/advantage_guidance.py as the method entry point. Its documented experiment command enables residual parameterization, seven imagined actions and action chunking. Code availability is an audit trail, not evidence that this notebook has reproduced the benchmark. No new training run was performed here.

6. Evidence: strong results with meaningful exceptions

The evaluation covers 20 OGBench and six OCBench tasks. The former uses play-style data; the latter uses narrower, demonstration-style data. These are different tests of extraction from a fixed dataset, not interchangeable samples of deployment conditions. Main results use ten seeds. The selected numbers below are task-family means, in percent, copied from v2 Appendix A.10, Table 5 (opens in a new tab). They are not a newly computed 26-task aggregate.

Family	Tasks	EDP	AG	AG − EDP, percentage points
Cube Single	5	99.0	94.9	−4.1
Cube Double	5	5.8	7.9	+2.1
Scene	5	73.8	86.0	+12.2
Puzzle 3×3	5	63.8	79.3	+15.5
Block Single	1	79.5	95.9	+16.4
Block Double	2	84.0	91.4	+7.4
Chamber Easy	3	54.6	73.3	+18.7

The AG 95% confidence intervals for these seven rows are respectively ±1.2, ±2.0, ±2.0, ±2.9, ±1.3, ±1.3 and ±1.6 percentage points. These mean-only differences are descriptive, not significance tests. Table 5 reports uncertainty; a difference of reported means does not supply a confidence interval on that difference. EDP is a selected comparator, not necessarily the strongest baseline in each row: CFGRL’s Scene mean is 92.9. A family average also hides failures: AG’s Scene T3 result is much lower than its results on the other Scene tasks. The authors’ aggregate success claim should therefore not be rewritten as universal taskwise dominance.

Section 7.1 adds a compute-dependent exception: sufficiently large best-of-N sampling slightly exceeds AG on OGBench, whereas AG remains stronger in the reported OCBench comparison. This matters because inference-time compute is part of the practical tradeoff.

The ablations connect the ingredients to the proposed failure modes: residual parameterization and action imagination improve responsiveness to advantage; improved responsiveness supports better policy extraction. Contrastive sampling is particularly useful on OCBench, while conditional sampling and CFG can be competitive on OGBench (Section 7.3). Appendix A.2 supports per-state normalization over batch or dataset normalization and finds diminishing improvements from additional imagined actions around seven. Very large guidance can hurt; changing label spacing or noise should be evaluated together with retuning its weight. Appendix A.4 also warns that simply replacing a naïve conditional architecture with a larger transformer does not fix the alignment problem. Main, best-of-N and sampling-method comparisons use ten seeds; other experiments generally use four (Appendix A.8). Label-noise and label-pair robustness should be read with their accompanying guidance-weight retuning. These are component comparisons within the authors’ setup, not a guarantee that each intervention helps every future task.

7. What remains uncertain, and what I would test

The most important dependency is critic reliability. A generative policy can become better at exploiting an estimated advantage without becoming better in the environment. Increasing guidance magnifies the desired preference and any systematic value error. Reporting evaluated advantage alone would miss that failure.

There are also three distinct approximations: fitting conditional fields, translating the clean-score argument into intermediate-time guidance, and integrating a finite number of steps. Better training does not automatically eliminate the latter two. Support restrictions and extrapolation remain relevant even if samples look plausible.

Research hypothesis, not a paper result: a learned dynamics model could evaluate candidate action chunks using predicted trajectories, then provide a richer quality signal to the same extraction machinery. That would introduce model error as well as critic error. I would first compare methods at equal policy capacity and inference cost, holding the quality estimator fixed; only then change that estimator. This note makes no factual claim about a specific “Flow-JEPA” implementation without reading and citing its source.

For biomedical control, the first experiment should be an offline simulator study with deliberately perturbed value estimates, measuring achieved return, constraint violations and sensitivity to guidance. It would test robustness, not establish clinical suitability. A clean follow-up would separate: does residual training improve controllability, does contrastive guidance improve decision quality at matched compute, and which failures arise from the critic rather than the sampler?

The durable contribution to retain is this distinction: a useful conditional generator needs both a learnable condition and a sampling rule whose preference matches the objective. AG offers a concrete connection between those two design questions, with empirical evidence and limitations that should travel together when the idea is reused.

Note provenance

Authoring tool: Codex. Model: not exposed by this run. Reasoning setting: not exposed by this run. Generated/updated: 2026-10-30.

Review: same-session source checks and deterministic mathematical tests. Review model: not exposed by this run. Review date: 2026-10-30. Independent review: no.

Human review: The owner reviewed the notes and companion outputs. This is not an independent scientific audit. Review date: 2026-10-30.