Alignment Forecasting: Predicting Misalignment From Training Data
Quick summary
arXiv:2609.35805v1 Announce Type: cross Abstract: Training a language model on data with a narrow flaw can sometimes make the model broadly misaligned. Inspecting the data at face value often does not settle whether it will emerge, and today it is caught only after training, by auditing the resulting model. To complement post-hoc audits, we introduce Alignment Forecasting: the task of predicting alignment failures before training. Given a target model, a fine-tuning dataset, and a failure mode such as deception or sycophancy, a forecaster outputs the probability that fine-tuning would meaningf
Key takeaways
- arXiv:2609.35805v1 Announce Type: cross Abstract: Training a language model on data with a narrow flaw can sometimes make the model broadly misaligned.
- Inspecting the data at face value often does not settle whether it will emerge, and today it is caught only after training, by auditing the resulting model.
- To complement post-hoc audits, we introduce Alignment Forecasting: the task of predicting alignment failures before training.
Why it matters
This model development creates a new option for users and a new testing obligation for developers. A fixed evaluation set comparing quality, cost and failure behavior is more useful than launch claims.
Member comments