Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding
Quick summary
arXiv:2604.02047v2 Announce Type: replace-cross Abstract: Speculative decoding accelerates large language model inference by drafting multiple candidate tokens and verifying them in a single forward pass. Candidates are organized as a tree: deeper trees accept more tokens per step, but adding depth requires sacrificing breadth (fallback options) under a fixed verification budget. Existing training-free methods draft from a single token source and shape their trees without distinguishing candidate quality across origins. We observe that two common training-free token sources -- n-gram matches c
Key takeaways
- arXiv:2604.02047v2 Announce Type: replace-cross Abstract: Speculative decoding accelerates large language model inference by drafting multiple candidate tokens and verifying them in a single forward pass.
- Candidates are organized as a tree: deeper trees accept more tokens per step, but adding depth requires sacrificing breadth (fallback options) under a fixed verification budget.
- Existing training-free methods draft from a single token source and shape their trees without distinguishing candidate quality across origins.
Why it matters
“Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding” should be evaluated beyond branding and benchmark scores. Its practical importance will emerge in task accuracy, latency, unit cost, safety and integration with real workflows.
Member comments