Topology-Aware Data Movement for Disaggregated GPU Inference
Quick summary
arXiv:2607.28633v2 Announce Type: replace-cross Abstract: Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly. When prefill and decode run on separate GPU pools, the KV cache must be transferred between them. For a 70B model this is 1.3 GB per request, exceeding 100 GB/s aggregate at production scale. Yet DistServe, Splitwise, and Mooncake all use uniform RDMA, ignoring that bandwidth between two GPUs varies by 72x depending on their physical relationship: 900 GB/s via NVLink 4.0 within a domain (1.8 TB/s on NVLink 5, widening the gap to
Key takeaways
- arXiv:2607.28633v2 Announce Type: replace-cross Abstract: Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly.
- When prefill and decode run on separate GPU pools, the KV cache must be transferred between them.
- For a 70B model this is 1.3 GB per request, exceeding 100 GB/s aggregate at production scale.
Why it matters
“Topology-Aware Data Movement for Disaggregated GPU Inference” exposes the compute, energy and supply-chain layer behind model competition. Capacity shifts can influence model costs, service availability and the ability of smaller companies to compete.
Member comments