We stopped guessing which prompt version a customer hit

Our customer facing assistant gave different answers to the same request across two parallel deployments and neither engineering nor CX could reproduce the bad path. Prompt templates lived in three code paths, a model switch had its own flag and the rollout record only told us which service build handled the call. Asking CX for another screenshot was getting embarrassing because the screenshot couldnt tell us the prompt, model, retrieval settings or rollout cohort behind the answer. We started versioning the prompts in Braintrust and the experiment diff showed which template revision dropped a constraint during cleanup. The quality drop lined up with one cohort. We rolled it back, reran the same cases against the previous version and when the lineage was there the rest was straightforward. The part I like most is that PM and CX can now hand us a trace link instead of trying to reconstruct the session in a ticket. How are other teams tying prompt versions, model changes, retrieval settings and rollout cohorts together without making every release record a manual chore? submitted by /u/ForeverOther5130 [link] [comments]