Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Abstract
On-policy distillation suffers from truncation collapse due to length inflation and repetition saturation, which StableOPD addresses through divergence constraints and rollout mixture distillation.
On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers. We identify a failure mode of OPD: as training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data. This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance. We attribute this problem to the interaction between student-induced data collection and the distillation objective, which implicitly favors long and repetitive rollouts. To address this issue, we propose StableOPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation. These together mitigate repetition-induced length inflation and further stabilize OPD training. Across multiple math reasoning datasets, our approach prevents truncation collapse, stabilizes training dynamics, and improves performance by 7.2% on average.
Community
Hi authors, thanks for the very interesting paper — I found the truncation and repetition analysis in OPD very useful.
I had a small implementation-detail question about EOS / stop-token handling. In the student–teacher setups you study, how did you handle cases where the student and teacher use different EOS token ids? For example, did you canonicalize or align EOS / stop tokens before computing teacher log-probs and the reverse-KL token advantages?
I’m asking because in our own OPD experiments we ran into a related issue: when the student and teacher had different raw EOS ids and we did not explicitly map them, we observed effects that looked quite similar — more non-termination, truncation, and repetitive rollouts. After mapping both EOS tokens to a shared semantic stop event before sampling / KL evaluation, as described in our recent TRB paper (arXiv:2605.31159), this issue seemed to largely go away in our setups.
This may be completely orthogonal to the mechanism you analyze, but I was curious whether EOS canonicalization was part of your implementation, or whether you tried an ablation with explicitly aligned EOS tokens. Thanks again for the nice work!
Get this paper in your agent:
hf papers read 2604.08527 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper