OpenAI Prompt Cache Diagnostics Is GA: What Gateway Operators Must Not Strip

Prompt cache misses have been one of the more frustrating debugging experiences on the OpenAI API. You have a long system prompt, you're paying for the 24-hour cache window, and usage.input_tokens_details.cached_tokens keeps coming back zero. No indication of what changed. No diff. Just uncached tokens and a bill that's higher than expected.

OpenAI's Prompt Cache Diagnostics, which became generally available on September 8, 2026 in the Responses API, changes that. The feature adds a structured prompt_cache_diagnostics field to responses when you request it, telling you specifically why a cache prefix didn't reuse. For teams running GPT-5.6 or GPT-6 Astra behind a gateway or router, there are two things worth knowing right away: what the API surface looks like, and what your proxy layer must not do to it.

How the diagnostic surface works

The mechanism is deliberately narrow. You pass prompt_cache_options.comparison_response_id with the id of a previous response whose prefix you expected the current request to reuse. OpenAI then compares the two requests and populates prompt_cache_diagnostics on the response if it finds a miss.

second = client.responses.create(
    model="gpt-6-astra",
    instructions=policy,
    input="Reply with exactly OK.",
    tools=[{"type": "function", "name": "get_date"}],
    prompt_cache_options={"comparison_response_id": first.id},
)

diagnostics = second.prompt_cache_diagnostics
if diagnostics is not None and diagnostics.type == "cache_miss":
    print(diagnostics.reason)                       # e.g. "tools_changed"
    print(diagnostics.comparison_reusable_tokens)   # tokens that *would* have hit
    print(diagnostics.cache_missed_tokens)           # tokens that actually missed

The reason field is the new operational information. The documentation gives tools_changed as an example, triggered when function tool definitions or their order change between requests. OpenAI's prompt caching guide has always noted that tool definitions are part of the rendered prefix — reorder them or add one mid-session and the entire prefix after that point is invalidated. What was missing until now was any way to confirm that's what happened.

Diagnostics are available for GPT-5.6 and later models in the Responses API. The minimum cacheable prefix is 1,024 tokens for GPT-5.6 and later. comparison_response_id must be from the same organization. Setting it does not change caching behavior for the current request — you can still reuse matching entries from other prior requests; the comparison is purely diagnostic.

The gateway/router implication

Here's where teams using a proxy layer need to check their setup. Many AI gateways, including middleware that normalizes request bodies before forwarding them to OpenAI, can drop unknown or non-standard fields. If your gateway strips prompt_cache_options before the request reaches the OpenAI API, prompt_cache_diagnostics never populates. You lose the diagnostic signal entirely without any error — the request succeeds, the cache misses silently, and you still don't know why.

The same risk applies to tool list handling. Gateway layers that sort or deduplicate tool definitions before forwarding — a seemingly innocuous normalization — will invalidate the cache prefix on every request where the canonical order differs from the upstream order. With diagnostics now available, this pattern becomes detectable: you'd see reason: "tools_changed" on consecutive requests even when your application code hasn't touched the tool list. If you're seeing that pattern, the transformation is happening in your routing layer, not in your application.

The practical check: inspect what your gateway forwards for a request that includes prompt_cache_options. If the field is absent from the forwarded request body, add it to the gateway's pass-through allowlist. If tool definitions are being reordered by your gateway's normalization logic, preserve insertion order.

The cross-provider picture

Anthropic's prompt caching doesn't have an equivalent diagnostic surface. When a Sonnet or Fable request misses the cache, usage.cache_read_input_tokens is zero, and that's the full signal. There's no structured reason code, no token-level comparison with a baseline request. For teams running a dual-provider setup — OpenAI models on one path, Anthropic models on another — this is a genuine observability gap. You can now triage cache misses on the OpenAI path with precision; on the Anthropic path you're still reasoning backward from before-and-after token counts.

Anthropic's cache TTL behavior is also different. Claude models default to a 5-minute cache window (extendable to 1 hour with cache_control: {"type": "ephemeral"}), while OpenAI models use a 5-to-10-minute window for most models with the 24-hour extended window available for non-ZDR orgs. A gateway routing between providers has to track which cache model applies and advise downstream instrumentation accordingly. Diagnostics make the OpenAI leg of that easier to instrument.

What this changes for operator monitoring

Before this release, a common approach was to compare input_token_count versus cached_tokens ratio over time and flag sessions where the ratio dropped below a threshold. That's a population-level signal — useful for alerting but not for root-cause. With prompt_cache_diagnostics, you can now attach structured miss reasons to individual requests and aggregate them: how many misses per day are tools_changed, how many are some other reason, and whether the rate changes after a deployment.

The diagnostics also compose with the Prompt Caching Dashboard that OpenAI released in August. The dashboard shows aggregate cache hit rates and cache reads per write at the org and project level. Diagnostics let you tie those aggregate numbers back to specific request patterns when the numbers look wrong.

What TheRouter users should check

If you're routing OpenAI requests through TheRouter, the gateway passes prompt_cache_options through to OpenAI without modification. Tool list ordering is preserved from the upstream request body. For the diagnostic workflow: pass comparison_response_id in your application-level requests the same way you would direct API calls. You can also use TheRouter's per-request log view to confirm that prompt_cache_options appears in the forwarded request body before debugging further upstream.

The more important operational question is whether your deployment pipeline has any layer between your application and TheRouter that transforms request bodies — a sidecar proxy, a load balancer with request mutation, or an API schema validator that drops unknown fields. Those are the layers to audit first when prompt_cache_diagnostics doesn't appear on responses.