Version Your Prompts

Store templates next to the grader. Review them like code. Name versions (billing-v12) in traces so a score drop can be blamed on a hash, not a feeling.

A version is everything that changes the next token:

• The spec text (or a hash of it)
• The few-shot file id
• The contract / schema id
• The model name (a silent vendor upgrade is not your adjectives)
• Sometimes temperature, JSON-mode flags, and the delimiter dialect

When finance or safety asks “what changed Tuesday,” you answer with prompt=billing-v12 model=tiny-1, not “we made it warmer.”

The dashboard is not the source of truth. Git is. A dashboard tweak with no PR is how you lose the only copy that worked. Rollback is checkout + deploy, not “paste yesterday from chat.”

If someone cannot PR a prompt, they cannot ship a prompt. That rule saves more incidents than any adjective.

Hash what you send

One sentence in the spec must change the id. If it does not, you are not hashing what you send. Hash the rendered spec bytes, or hash the files that render() reads. Do not hash a nickname someone typed in a form while the real file drifted.

Log prompt_id on every LLM span (LLM logging lesson). Log model name next to it. A score drop with no id is a ghost.

A/B is two ids on a frozen eval, then a shadow on live traffic. Winner needs a better score and no new safety fails. Do not crown a winner from one founder chat.

What belongs in the PR

• The spec diff (readable English)
• Why (failing cluster, not “vibes”)
• Eval before / after on the named suite
• Token count if the prompt grew
• Confirmation that last few-shot is still a legal object
• Confirmation that disabled tools are not in shots

If the PR cannot show a score, it is not a prompt change. It is a hope.

Run to execute this in your browser. Nothing is sent to a server.

What printed: v1 and v2 differ because “Be nicer” is a real change. The trace carries prompt_id and model. Hashing the same three strings again matches v1. If “nicer” did not change the id, you are not hashing what you send.

Walkthrough: warmer, new hash

“Be nicer” changes one sentence. The hash must change. Wednesday’s score drop carries a prompt_id and a model name. You check out the previous tag and the suite recovers. If the dashboard had been the source of truth, the old poem is gone. If you hashed a nickname instead of the rendered bytes, both versions collide and you cannot rollback.

A/B is two ids, one frozen suite, then a shadow. Winner needs a better score and no new safety fails. One founder chat is not a winner.

What goes wrong if you skip this

You lose the only copy that worked. Score drops have no name. Silent model upgrades look like your adjectives. People edit prompts in a UI with no PR. Incidents cannot be bisected.

A version is not a mood. It is the bytes that change the next token: spec, shots, contract, model name, JSON-mode flags, delimiter dialect. Hash those bytes. Log the hash on every span. When finance asks what changed Tuesday, you read a field, you do not hold a séance.

Dashboard editors feel fast until they are the only copy. Git plus a PR is slower in the afternoon and faster in the incident. Rollback is checkout. A/B is two hashes on one frozen suite, then a shadow. Crowning a winner from a founder chat is how you ship a key dump with better manners.

If a sentence in the spec does not change the id, you are hashing a nickname. Hash render() output, or hash the files render() reads, and make that rule a unit test: mutate one character, id changes; mutate nothing, id stable.

Common mistakes

Log prompt_id and model name on every span. Rollback is checkout of the tag that last passed the suite. A/B is two hashes, one frozen eval, then a shadow. If “be nicer” does not change the hash, your hasher is lying. Write a unit test that mutates one character of the spec and asserts the id moved. That test is the versioning lesson in one function.

How agents use this

Log prompt_id on every LLM span. Rollback is checkout + deploy. A/B is two ids plus a frozen eval plus a shadow. The dashboard is not the source of truth. Git is. Blame a hash, not a feeling. Hash the rendered bytes. Include shots, contract, and model name in what you log. If someone cannot PR a prompt, they cannot ship a prompt. That rule saves more incidents than any adjective, and it is the whole versioning lesson. The dashboard is a viewer. Git is the store. A score drop without a hash is a ghost.

Tip:If someone cannot PR a prompt, they cannot ship a prompt. That rule saves more incidents than any adjective.