FeatCal: Feature Calibration for Post-Merging Models
Quick summary
arXiv:2605.13030v2 Announce Type: replace-cross Abstract: Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This
Key takeaways
- arXiv:2605.13030v2 Announce Type: replace-cross Abstract: Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts.
- We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input.
- Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift.
Why it matters
This development shows AI moving deeper into everyday software. Productivity potential should be weighed against price, data permissions, exportability and the preservation of human control.
Member comments