Prompt Caching

Cache long static context and pay less per request

Prompt caching is most effective when you keep a stable prefix (system prompt, policy text, retrieved context) and move dynamic user input to the end.

Reference payload

Use this baseline request shape and adapt model, provider order, and token limits to your workload.

request.json

{
  "messages": [
    {
      "role": "system",
      "content": [
        {"type": "text", "text": "long policy text"},
        {"type": "text", "text": "long KB chunk", "cache_control": {"type": "ephemeral"}}
      ]
    },
    {"role": "user", "content": "answer with citations"}
  ]
}

Configuration examples

TheRouter.ai keeps request semantics consistent across providers, so you can tune behavior without rewriting your app layer.

TypeScript

const payload = {
  model: "anthropic/claude-sonnet-4.5",
  messages: [
    {
      role: "system",
      content: [
        { type: "text", text: kbChunk },
        { type: "text", text: policies, cache_control: { type: "ephemeral", ttl: "1h" } },
      ],
    },
    { role: "user", content: userPrompt },
  ],
};

Production note

Operate with guardrails

Cache writes can be more expensive on some providers than cache reads. Measure hit rates before rolling out globally.

Use the activity feed and usage exports to validate that these settings improve reliability and cost in your real traffic mix.

Responses to a request that carried cache_control breakpoints include x-router-prompt-cache, which reports how the gateway handled the breakpoints (not cache hits or billing): forwarded means every breakpoint was sent upstream unchanged; stripped means at least one was removed before the upstream call. Possible causes of stripped: the model or provider is not on the prompt-caching allowlist; the request is not eligible for caching (for example it uses managed capabilities, skills or plugins); cache pricing for the route is not loaded; the provider does not support prompt caching on TheRouter; or a breakpoint could not be carried through the /v1/messages to OpenAI-format translation. On non-streaming responses the header describes the attempt that produced the response. On streaming responses it describes the first dispatched attempt, because stream headers are sent before the upstream answers: a later failover is not reflected, and if the provider rejects the breakpoints and the gateway retries without them, a stream can report forwarded although the breakpoints were removed. Response-cache hits carry no header.

Latency & Performance

Uptime Optimization