Versioning the prompt finally made our rollout chart mean something

We rewrote a prompt and the overall answer score rose, so the first chart looked great. Then we split results by experience level. Newcomers got clearer steps while expert customers got longer answers that buried the usefulness. A model update had also landed that afternoon, which meant the aggregate chart couldn’t tell us which change moved quality. We versioned the prompt in Braintrust with immutable versions, ran both versions against a fixed dataset snapshot and attached cohort metadata to the experiment. The difference showed that the instruction change caused the extra detail for experts, while the model update helped citation accuracy across both groups. Prompt lineage gave us a clean comparison instead of trying to reconstruct the afternoon from commits and chat threads. I’m curious what metadata people attach to prompt experiments so an aggregate gain doesn’t hide a meaningful regression for experienced customers? submitted by /u/ImpossibleFood8242 [link] [comments]