How to keep track of prompt changes?

I'm starting to believe that keeping production prompts in the codebase is one of those decisions that feels harmles until you have to explain a quality drop. At the moment, we’ve got prompts scattered all over the place. Some are in config files, some live in helper functions and a few are buried who knows where. Then someone tweaks the prompt, someone else changes the model and a few days later the workflow starts behaving differently. Half the investigation is just figuring out what changed. Versioning the prompt is only a small piece of the problem. What we've struggled with is understanding why quality moved in the first place. By the time someone notices the change, there have been a few deployments, a model update and maybe an eval refresh. Looking back at all of that and working out which change mattered is a lot harder than keeping old prompt versions around. Vibes-based prompt edits is all fun and games during the MVP phase, but when there are customers using it, that's when it becomes serious. I feel like that's when boring things like prompt history and good evals become really important. submitted by /u/Fair-Fruit-646 [link] [comments]