Cache the Stable Prefix, Not Today’s Request

A long AI prompt often contains two very different things: the material that defines the job every time, and the material that is new for this one request. A support assistant may carry a policy, product taxonomy, tool definitions, examples, and output rules on every call; only the customer message changes. If that stable material is sent in the same order each time, prompt caching can reuse the model’s prior work on the shared beginning of the request. If a timestamp, random ID, or rewritten instruction appears near the front, the cache cannot recognize the prefix it already processed.

The practical technique is simple: make the reusable prefix intentionally boring and stable, then put per-request data at the end. It is a prompt-layout decision, not a feature you enable and forget. OpenAI says cache reuse requires the entire rendered prefix to match, including applicable instructions, tool definitions, and prior context.^[1] Anthropic likewise says a cache entry covers the full prefix through the chosen breakpoint; changing anything at or before that point creates a different prefix.^[2]

Separate the durable kit from the live request

Start by classifying every piece of an API request by how often it changes.

Durable kit: system or developer instructions, tool schemas, fixed policy, output rules, reference examples, and a stable knowledge packet. Keep this material ordered consistently. Do not regenerate it from a template that inserts a run ID or the current date.

Live request: the user’s latest question, a document to extract, a ticket, a query result, or values specific to this run. Put it after the stable kit. It is supposed to change; do not try to cache it.

This distinction is especially useful for repeated extraction, classification, review, and agent calls that carry large tool definitions. OpenAI describes prompt caching as reusing computation for matching prefixes, while Anthropic identifies repeated tasks, large background context, many examples, and long conversations as useful cases.^[1][2] The gain is not a better answer by itself. It is less repeated input processing when the request really shares context.

Copy this stable-prefix checklist

Use this during prompt construction or an API integration review:

GOAL: Requests in this group should share one cacheable beginning.

1. BUILD THE DURABLE KIT FIRST
   - Fixed system/developer instructions
   - Versioned policy and output contract
   - Tool definitions in a deterministic order
   - Stable examples and reference context

2. MARK THE BOUNDARY
   - Place the provider's cache breakpoint after the last stable block.
   - Keep any cache key stable for requests that should share routing/accounting.

3. APPEND LIVE DATA AFTER THE BOUNDARY
   - Current user message or document
   - Timestamp, request ID, locale, or experiment assignment
   - Fresh retrieval results and tool outputs

4. MEASURE, THEN ADJUST
   - Record cached-input/read tokens and cache writes.
   - Compare hit rate for one real request group, not all traffic combined.
   - Investigate any sudden miss after a prompt or tool-schema release.

5. CHANGE DELIBERATELY
   - Version the durable kit.
   - Expect the first request after a real prefix change to miss or write anew.
   - Do not mutate an old prefix just to preserve a hit.

For a ticket classifier, the durable kit might contain the classification policy, allowed labels, JSON contract, and three examples. The live request contains the ticket body and its metadata. If your implementation supports an explicit breakpoint, set it immediately after the examples—not after a live block that contains a timestamp. Anthropic documents this exact failure mode: a breakpoint after changing content can result in a fresh cache write every request instead of a cache read.^[2]

Stabilize more than the text

A cache miss is often caused by an invisible change in request assembly. Tool definitions, response schemas, message ordering, and model-side instructions can be part of the rendered prefix. OpenAI explicitly includes tool definitions and developer messages in that context.^[1] Treat them like a small API surface: keep them versioned, sorted, and tested.

This does not mean never changing the instructions. A policy correction is more important than a cache hit. It means make the change on purpose, deploy the new version as a coherent prefix, and expect the cache to warm again. If you personalize instructions by customer, segment the traffic so each group has a genuinely reusable common prefix; do not put a unique customer profile before shared policy unless that personalization is needed for correctness.

Failure mode: optimizing a cache you do not actually hit

Prompt caching is opportunistic. A matching prefix may still miss because it is too short for a provider’s threshold, expired, routed differently, or changed in a field your code did not consider. OpenAI notes that cache entries are not permanent and that cache reuse depends on a matching entry being available on the relevant machine.^[1] Anthropic’s default cache lifetime is five minutes, measured from the start of the request that reads or writes it.^[2]

So do not declare success because caching is configured. Inspect each provider’s usage fields or diagnostics and look at cached reads over representative traffic. Also avoid caching sensitive material merely because it is stable: follow the provider’s retention and data-handling documentation, and keep secrets out of prompts whenever the workflow does not require them.