Muon Is Theoretically Wrong For Convolutions, But Empirically Effective
Quick summary
arXiv:2610.07103v1 Announce Type: cross Abstract: Muon, an optimizer known for its efficiency, has a clear interpretation for matrix-valued updates, but convolutional kernels are stored as four-dimensional tensors. Standard implementations reshape these tensors into matrices, a shortcut which breaks the theoretical understanding behind Muon. To investigate this, we formalize the corresponding optimization objective directly in convolutional operator geometry and introduce Convolutional Newton-Schulz (Conv-NS), which approximates the polar factor in this geometry while preserving kernel support
Key takeaways
- arXiv:2610.07103v1 Announce Type: cross Abstract: Muon, an optimizer known for its efficiency, has a clear interpretation for matrix-valued updates, but convolutional kernels are stored as four-dimensional tensors.
- Standard implementations reshape these tensors into matrices, a shortcut which breaks the theoretical understanding behind Muon.
- To investigate this, we formalize the corresponding optimization objective directly in convolutional operator geometry and introduce Convolutional Newton-Schulz (Conv-NS), which approximates the polar factor in this geometry while preserving kernel support
Why it matters
The importance of “Muon Is Theoretically Wrong For Convolutions, But Empirically Effective” will be measured by what changes in practice. User behavior, access conditions, verifiable performance and responsible-use outcomes are the signals worth following.
Member comments