K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments
Quick summary
arXiv:2609.12808v2 Announce Type: replace Abstract: Unlearning benchmarks such as TOFU and MUSE certify forgetting by reading the model's final answer, where a model that refuses to answer already counts as having forgotten. We show that this model-level certificate does not transfer once the model is deployed as an agent. We introduce K-Bench, a benchmark that scores LLM unlearning under agentic deployment. K-Bench inspects all six channels a ReAct agent exposes, including its chain-of-thought (CoT), tool calls and tool observations, and elicited summary. A query counts as leaked if the secre
Key takeaways
- arXiv:2609.12808v2 Announce Type: replace Abstract: Unlearning benchmarks such as TOFU and MUSE certify forgetting by reading the model's final answer, where a model that refuses to answer already counts as having forgotten.
- We show that this model-level certificate does not transfer once the model is deployed as an agent.
- We introduce K-Bench, a benchmark that scores LLM unlearning under agentic deployment.
Why it matters
“K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments” highlights the need for repeatable measurement rather than a single impressive demonstration. Independent validation across datasets and clearly stated limitations determine whether a result can guide product decisions.
Member comments