False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift
Quick summary
arXiv:2610.01535v1 Announce Type: cross Abstract: Safety routers send each request to one of several models and are judged against the best single model. A major routing benchmark picks that comparator on the evaluation data. In the benchmark's own setting this is harmless, but under distribution shift it is not. On HELM Safety the selection cost is 0.003-0.030 of harm under random splits and 0.045-0.113 under held-out categories, comparable to the whole deficit attributed to routing, with its direction holding under either published judge alone. It rises seven- to ninefold on AgentDojo when s
Key takeaways
- arXiv:2610.01535v1 Announce Type: cross Abstract: Safety routers send each request to one of several models and are judged against the best single model.
- A major routing benchmark picks that comparator on the evaluation data.
- In the benchmark's own setting this is harmless, but under distribution shift it is not.
Why it matters
“False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift” shows why AI risk cannot be reduced to answer accuracy. Access controls, logging, human approval and incident response need to be designed into the workflow from the start.
Member comments