Removing spurious minima for planar features by skip connections
Quick summary
arXiv:2610.01728v1 Announce Type: cross Abstract: Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student
Key takeaways
- arXiv:2610.01728v1 Announce Type: cross Abstract: Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models.
- We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting.
- This provides a simple model for studying essential aspects such as feature learning and overparameterization.
Why it matters
“Removing spurious minima for planar features by skip connections” highlights the need for repeatable measurement rather than a single impressive demonstration. Independent validation across datasets and clearly stated limitations determine whether a result can guide product decisions.
Member comments