Constitutional On-Policy Safe Distillation
Quick summary
arXiv:2606.03089v3 Announce Type: replace-cross Abstract: On-policy self-distillation (OPSD) has emerged as an efficient post-training paradigm by using a teacher conditioned on privileged information to provide dense token-level supervision. Prior work has shown that OPSD can collapse in verifiable reasoning tasks, while safety alignment differs in that it is guided by high-level constitutions rather than explicit target answers. However, pilot studies reveals that safety OPSD nonetheless suffers from severe collapse, where constitutional conditioning contracts the teacher distribution toward
Key takeaways
- arXiv:2606.03089v3 Announce Type: replace-cross Abstract: On-policy self-distillation (OPSD) has emerged as an efficient post-training paradigm by using a teacher conditioned on privileged information to provide dense token-level supervision.
- Prior work has shown that OPSD can collapse in verifiable reasoning tasks, while safety alignment differs in that it is guided by high-level constitutions rather than explicit target answers.
- However, pilot studies reveals that safety OPSD nonetheless suffers from severe collapse, where constitutional conditioning contracts the teacher distribution toward
Why it matters
“Constitutional On-Policy Safe Distillation” may affect what data AI products can use and where accountability sits. Product teams should watch compliance duties, rights holders should watch enforcement, and users should watch transparency and appeal mechanisms.
Member comments