On the policy gradient foundations of group relative policy optimization: credit assignment, gradient sparsity, and rank collapse
A rigorous derivation of GRPO from first principles of the policy gradient theorem.
Group Relative Policy Optimization (GRPO) eliminates the learned critic in PPO by using the mean reward of grouped rollouts as a baseline. We provide a rigorous derivation of GRPO from first principles of the policy gradient theorem, revealing a fundamental credit assignment failure: under output-only reward, every token in a rollout receives identical advantage, collapsing token-level credit to a single scalar. We prove this induces gradient sparsity that intensifies over training, and demonstrate empirically via SVD analysis of GRPO gradients on Nemotron-4B/GSM8K that the gradient matrix has effective rank $\approx 2$ regardless of group size $R \in \{2,4,8\}$. We formalize this as an intrinsic rank-2 structure arising from the zero-sum constraint on advantages and derive conditions under which GRPO's baseline is optimal. Our results characterize when GRPO's simplicity is theoretically justified and identify the credit assignment bottleneck as the key limitation for multi-step reasoning.
Latest publications
Critique-guided distillation for robust reasoning via refinement
A training framework that decouples critique consumption from critique generation.
ICMLAsk, don’t judge: Binary questions for interpretable LLM evaluation and self-improvement
A framework that decomposes evaluation criteria into atomic binary questions and aggregates the resulting verdicts.
ICMLSAFARI: Scaling long horizon agentic fault attribution via active investigation
A framework that replaces linear context loading with a tool-augmented diagnostic loop.
ICML