Form Over Content In Gradient-Based Data Attribution Methods
Quick summary
arXiv:2609.19589v1 Announce Type: cross Abstract: Data attribution methods using gradient similarity are widely used to analyze and select training data for large language models, but what gradient similarity actually measures is debated. Some interpret it as identifying task-relevant skills, while other work reports that surface form is the main factor. We resolve this debate for supervised fine-tuning examples by varying task and answer format independently. Specifically, we render benchmarks in different answer formats, such that datasets can share a task without a format or a format withou
Key takeaways
- arXiv:2609.19589v1 Announce Type: cross Abstract: Data attribution methods using gradient similarity are widely used to analyze and select training data for large language models, but what gradient similarity actually measures is debated.
- Some interpret it as identifying task-relevant skills, while other work reports that surface form is the main factor.
- We resolve this debate for supervised fine-tuning examples by varying task and answer format independently.
Why it matters
The importance of “Form Over Content In Gradient-Based Data Attribution Methods” will be measured by what changes in practice. User behavior, access conditions, verifiable performance and responsible-use outcomes are the signals worth following.
Member comments