Attention-Weighted Value Projection for KV-Cache Compression
Quick summary
arXiv:2604.11501v2 Announce Type: replace-cross Abstract: Rank reduction discards dimensions; quantization keeps them at lower precision. Comparing the two requires a choice of what compression should preserve. For attention values, we study reconstruction of the attention output rather than reconstruction of the values alone. With fixed attention weights, the optimal orthogonal rank-$r$ projection uses the leading eigenvectors of $V^\top\alpha^\top\alpha V$, and its error is exactly the discarded eigenvalue sum. We extend this objective to calibration datasets and grouped query attention, and
Key takeaways
- arXiv:2604.11501v2 Announce Type: replace-cross Abstract: Rank reduction discards dimensions; quantization keeps them at lower precision.
- Comparing the two requires a choice of what compression should preserve.
- For attention values, we study reconstruction of the attention output rather than reconstruction of the values alone.
Why it matters
The value of this work lies as much in how it was tested as in the claim itself. Sample design, baselines, uncertainty and replication help separate a laboratory result from real-world impact.
Member comments