Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
Quick summary
arXiv:2609.27085v1 Announce Type: cross Abstract: As serving capacity demand surpasses that of training, serving efficiency becomes increasingly important. Prefill-decode (P/D) disaggregation improves serving efficiency through specialization and isolation of the two phases. These benefits rest on a static partitioning. Phase demand, however, is not static. We observe that in a large LLM fleet the ratio of uncached input to output tokens has peak-to-mean ratios up to 4.7x at minute timescales, and that in a public agentic trace the hourly ratio spans a median 24.5x within a single day, while r
Key takeaways
- arXiv:2609.27085v1 Announce Type: cross Abstract: As serving capacity demand surpasses that of training, serving efficiency becomes increasingly important.
- Prefill-decode (P/D) disaggregation improves serving efficiency through specialization and isolation of the two phases.
- These benefits rest on a static partitioning.
Why it matters
The importance of “Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving” will be measured by what changes in practice. User behavior, access conditions, verifiable performance and responsible-use outcomes are the signals worth following.
Member comments