Exact Attention Sensitivity and the Geometry of Transformer Stability
Quick summary
arXiv:2602.18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation. Our main result is the exact identity $\|J_\tau(u)\|_{\infty\to1}=\theta(p)/\tau$ for the Jacobian $J_\tau(u)$ of the tempered softmax $u\mapsto\mathrm{softmax}(u/\tau)$, where $\theta(p)=4\max_{S\subseteq[L]}p(S)(1-p(S))$ measures how evenly the attention distribution can be bisected rather than how concentrated it is. We combine this identity with a block-$\infty$/RMS norm under which row-stochastic attention mixing is nonexpa
Key takeaways
- arXiv:2602.18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.
- Our main result is the exact identity $\|J_\tau(u)\|_{\infty\to1}=\theta(p)/\tau$ for the Jacobian $J_\tau(u)$ of the tempered softmax $u\mapsto\mathrm{softmax}(u/\tau)$, where $\theta(p)=4\max_{S\subseteq[L]}p(S)(1-p(S))$ measures how evenly the attention distribution can be bisected rather than how concentrated it is.
- We combine this identity with a block-$\infty$/RMS norm under which row-stochastic attention mixing is nonexpa
Why it matters
“Exact Attention Sensitivity and the Geometry of Transformer Stability” illustrates how changes in the AI ecosystem can affect products, workflows and user expectations together. Its lasting significance depends on measurable adoption, cost and safety outcomes.
Member comments