Does On-Policy Distillation for Safety Pose Backdoor Risks?
Quick summary
arXiv:2610.07654v1 Announce Type: cross Abstract: On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models. Recent studies further explore OPD as a tool for improving large language model safety with promising results. However, these approaches typically assume that the teacher and training data are trustworthy. In this paper, we uncover an overlooked threat to OPD for safety: a safety-aligned but backdoored teacher can propagate its hidden malicious behavior to an initially clean student. Under our threat m
Key takeaways
- arXiv:2610.07654v1 Announce Type: cross Abstract: On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models.
- Recent studies further explore OPD as a tool for improving large language model safety with promising results.
- However, these approaches typically assume that the teacher and training data are trustworthy.
Why it matters
“Does On-Policy Distillation for Safety Pose Backdoor Risks?” may affect what data AI products can use and where accountability sits. Product teams should watch compliance duties, rights holders should watch enforcement, and users should watch transparency and appeal mechanisms.
Member comments