arXiv Artificial Intelligence

Attn-QAT: 4-Bit Attention With Quantization-Aware Training

2 hours ago 11.08.2026 07:00 👁 1 views
Attn-QAT: 4-Bit Attention With Quantization-Aware Training

Quick summary

arXiv:2603.00040v3 Announce Type: replace-cross Abstract: Achieving reliable 4-bit attention is a prerequisite for end-to-end FP4 computation on emerging FP4-capable GPUs, yet attention remains the main obstacle due to FP4's tiny dynamic range and attention's heavy-tailed activations. This paper presents the first systematic study of 4-bit quantization-aware training (QAT) for attention. We find ``drop-in'' QAT -- naively combining an FP4 forward pass with high-precision Flash Attention (FA)-style backward pass -- leads to training instability. We identify two key principles for stable FP4 att

Key takeaways

  • arXiv:2603.00040v3 Announce Type: replace-cross Abstract: Achieving reliable 4-bit attention is a prerequisite for end-to-end FP4 computation on emerging FP4-capable GPUs, yet attention remains the main obstacle due to FP4's tiny dynamic range and attention's heavy-tailed activations.
  • This paper presents the first systematic study of 4-bit quantization-aware training (QAT) for attention.
  • We find ``drop-in'' QAT -- naively combining an FP4 forward pass with high-precision Flash Attention (FA)-style backward pass -- leads to training instability.

Why it matters

“Attn-QAT: 4-Bit Attention With Quantization-Aware Training” highlights the need for repeatable measurement rather than a single impressive demonstration. Independent validation across datasets and clearly stated limitations determine whether a result can guide product decisions.

Kaynak sitede devamını oku: arXiv Artificial Intelligence ↗