HEART: Exploiting Head Heterogeneity in Sparse Attention for Video Diffusion

Quick summary

arXiv:2605.14513v2 Announce Type: replace-cross Abstract: Sparse attention accelerates video diffusion by allowing each attention head to focus on only a small subset of interactions. Existing methods already construct head-specific sparse patterns conditioned on the input. However, we find that these heads also differ in two less obvious but practically important ways. First, for some heads, sparse attention masks remain stable across many denoising steps, whereas others change rapidly. Second, heads differ substantially in their sensitivity to sparsification: applying the same threshold can