OpenAI Prompt Cache Retention Defaults to 24h: GPT-5.5 and API Routing Impact

On May 29, 2026, OpenAI quietly changed how extended prompt cache retention works for organizations using GPT-5.5, GPT-5.4, GPT-5.2, and the broader GPT-5-series on its API. For operators without Zero Data Retention (ZDR) enabled, prompt_cache_retention now defaults to 24h instead of in_memory. For GPT-5.5 and future models, 24h is the only supported retention mode — there is no in_memory option.

If you route traffic to OpenAI through an AI gateway and have not yet reviewed how this interacts with your data policy and cost model, it is worth a few minutes now rather than a surprise at invoice time.

What extended prompt cache retention actually means

OpenAI's prompt cache works automatically for prompts of 1,024 tokens or more by routing API requests to machines that recently processed an identical prompt prefix, avoiding redundant prefill computation. The original implementation used an in-memory policy: cached prefixes stayed alive for 5–10 minutes of inactivity, with a maximum one-hour window, held only in volatile GPU memory.

Extended retention (up to 24 hours) keeps those key/value tensors alive much longer by offloading them to GPU-local storage when memory pressure is high. Critically, only the key/value tensors are persisted to storage — the original prompt text is never written to disk and stays only in memory. OpenAI explicitly frames this as a storage efficiency optimization, not a data persistence change.

Cache hit pricing remains the same under both policies. The shift to 24h as the default is purely about improving cache hit rates across request spikes spread over hours, not minutes.

Which models are affected and in which direction

The behavior splits by model generation:

If your organization has ZDR enabled, the default for older models remains in_memory. ZDR organizations prefer the shorter window by default to limit the persistence footprint of cached tensors. For GPT-5.5 and newer, even ZDR organizations get 24h — there is no override.

Why this matters for API routing operators

Extended cache retention changes the cost calculus in two ways that matter at gateway or multi-tenant operator scale.

1. Cache hit rates improve significantly for bursty or scheduled workloads. Agent pipelines, batch evaluation jobs, and scheduled report generation all tend to reuse the same system prompt prefix across many requests — but often spread those requests over hours, not minutes. Under the old in_memory policy, the cache was cold by the second batch. Under 24h, the prefix stays warm. For workloads with long, stable system prompts and variable request cadence, this can meaningfully reduce effective input token costs — OpenAI cites up to 90% cost reduction on cache hits and 80% latency reduction.

2. The prompt_cache_key parameter becomes more useful. OpenAI's caching routes requests by a hash of the prompt's initial prefix (roughly the first 256 tokens). When request volume for the same prefix and prompt_cache_key combination exceeds approximately 15 requests per minute, overflow is routed to additional machines, reducing cache effectiveness. Setting prompt_cache_key on your gateway or SDK config lets you influence that routing, improving hit rates for high-volume prefixes. With 24h retention, a well-chosen key helps warm a cache that stays available all day.

3. Data-sensitive tenants need explicit configuration. If you operate a multi-tenant API proxy and some tenants have strict data residency or persistence requirements, you may need to explicitly set prompt_cache_retention: in_memory per-request for those tenants on models that still support both options. Newer GPT-5.5 and future models will not honor in_memory even if you send it — route those tenants to models that still support short-window retention, or obtain ZDR for the relevant organization.

What to configure at the gateway layer

For operators using an AI gateway to route to OpenAI's Responses API or Chat Completions API, the actionable changes are:

{
  "model": "gpt-5.5",
  "input": "...",
  "prompt_cache_retention": "24h"
}

This is now the default for non-ZDR organizations; passing it explicitly makes intent clear and future-proofs against policy changes.

For tenants where you want to minimize caching duration on models that still support in-memory retention:

{
  "model": "gpt-5.4",
  "input": "...",
  "prompt_cache_retention": "in_memory"
}

The prompt_cache_key field is available on Responses.create (and chat.completions.create) and helps stabilize cache routing when many tenants share the same long system prompt:

{
  "prompt_cache_key": "tenant-shared-system-v12"
}

What to watch

OpenAI notes the 24h default is enabled "for organizations without ZDR enabled" for the GPT-5 series. It is reasonable to expect that as future model generations ship, the in_memory option will continue to narrow or disappear. Operators who have built cost models assuming short-lived cache windows should update projections accordingly — the caching benefit is now structural, not opportunistic.

Try it

Use TheRouter as the policy layer in front of OpenAI when teams need one OpenAI-compatible endpoint, unified billing, and tenant-aware routing decisions. Review your OpenAI provider setup and quickstart before changing defaults in production. Track cached_tokens, cache-hit-sensitive latency, and tenant data-policy exceptions after rollout; without those three checks, a 24h cache default is only a cost assumption, not an operating control.