Prompt Caching

Prompt caching (prefix cache) means: if the start of the prompt is the same as a recent call, the vendor may charge less for those input tokens and skip some prefill work. You still send the messages. They still count toward the window. You may pay a cheaper rate for the repeated head.

This is not magic memory. The model does not “remember last Tuesday.” The bytes at the front must be byte-stable across calls. Shuffle the system prompt, inject the current time at the top, or reorder tools, and you miss the cache. A 5% hit rate is not a strategy. It is a random discount.

Your GPU KV cache (Transformers track) is per running sequence while a generation is in flight. Vendor prompt cache is a billing and prefill feature across calls, minutes later, maybe from another machine. Related idea, different product. Do not tune one hoping to fix the other.

Stable prefix, volatile tail

Put the stable bits first:

1. Long pinned spec (versioned, not edited per request)
2. Tool docs that rarely change
3. Then the changing user turn, last observation, retrieved chunks, timestamps

If you must include “now,” put it last, or in the user message, not as line one of system. If you A/B two specs, you will split the cache — that is expected; measure both.

Minimum size and TTL (how long the vendor keeps the prefix) are vendor-specific. Some require a marker or a header. Some only cache above a token threshold (for example, a long spec). Read the current docs when you implement. This lesson is the byte-stability rule, which does not change when the header name does.

Run to execute this in your browser. Nothing is sent to a server.

Same user, same order: prefix matches (True). Clock as the first message: match breaks (False). New job id at the end: the first 40 characters of the blob still match, because spec and tools came first. That is the win: many tickets share a spec prefix; only the tail changes.

If you put now=12:01 first, every minute is a new prefix. You paid for a clock to miss the discount. Put the clock last if you need it at all. Many agents do not need a clock in the prompt; a tool get_time is an observation in the tail.

What belongs in the prefix

Good: a 1–2k token spec you version as spec@2026-03-01, a stable tool list, a short legal example. Bad: per-user personalization at line one, shuffled tool order, retrieved chunks (they change every query), “session so far” dumps.

Tool docs should be the enabled set, and that set should not shuffle. If you sort tool names alphabetically, do it every time. Random order from a dict iterate is a cache miss on some Python versions and a heisenbug in your bill.

Measure hit rate

Log cached vs uncached input tokens when the vendor provides them. Hit rate = cached / (cached + uncached) on the prefix-eligible portion. If you cannot log it, you cannot claim caching as a savings plan. A cache that never hits is complexity for nothing.

Do not build a second cache in front of the vendor unless you have a measured miss. Your own identical-prompt cache (same bytes → same completion) is a different, useful idea for deterministic steps at temperature 0 — and it has consistency issues if the model or spec changes. Version the key with model + spec id.

Hit rate without a threshold is a vanity metric. If the vendor only caches prefixes longer than N tokens, a 200-token spec will never hit. Lengthen the stable spec (tool docs count) or stop claiming cache as a plan. Measure cached_tokens / prompt_tokens on traces that share the spec, not on the first call of the day (cold cache).

TTL: if the vendor drops the prefix after 5 minutes of silence, a low-traffic agent will never hit. Batching and prefix stability both matter. A burst of tickets after a deploy is the moment you should see hits — if you shuffled tools in the deploy, you will not.

Clock and request id belong in the tail, or in a tool observation. A/B specs split the cache on purpose; name the spec id so you do not call that split a regression. Measure hit rate per spec id.

What goes wrong

• Timestamp or request id at the top of system.
• Reordering tools every call.
• Per-user “you are talking to Maya” as the first line instead of a later metadata line.
• Assuming cache = memory. The model still only sees this prompt.
• Not measuring. Then a packing change silently kills the discount.

How agents use this

Pin the spec. Do not shuffle tools. Do not put the current time in the first message. Put dynamic stuff last. Caching is another reason the context-engineering lesson will say: stable prefix, volatile tail.

Unit-test build_context: the first N characters of the serialized prefix are identical across two tickets that share a spec. If a developer adds str(datetime.now()) to system, the test fails. That test is cheaper than a month of missed cache.

Sort tool names in code, every time. Put request ids, clocks, and user text after the prefix. Log cached vs uncached when the vendor provides the split. If they do not, you cannot manage cache; you can still keep the prefix stable for prefill speed on some hosts.

Note:GPU KV cache is per running sequence. Vendor prompt cache is a billing feature across calls. Related idea, different product. Context packing (later) is how you keep the tail small either way.