OpenAI Splits the Rate-Limit 429: What 'slow_down' vs 'server_is_overloaded' Mean for Your Retry Policy

On September 2, OpenAI pushed a small changelog entry that looks routine at first glance. It isn't. The change splits what used to be a single ambiguous HTTP 429 into two semantically different error responses — and the split matters a lot for gateway retry logic and for how you route traffic under load.

The old behavior: one error code for two completely different problems

Before this change, if your API call came back with a 429, you didn't know whether you were hitting a rate limit (your traffic is too fast), or hitting model saturation (OpenAI's side is temporarily overloaded). Both conditions returned the same HTTP status with the same undifferentiated error body.

This forced most retry implementations into a single exponential-backoff strategy regardless of root cause. Throttle your own throughput because you're hammering a rate limit? Correct. Throttle your own throughput because OpenAI's model inference layer is briefly overwhelmed and you should retry at your original rate shortly? That makes you slower than necessary and compounds the problem by holding work in a queue longer than needed.

Practically, every open-source AI gateway wrapper — LiteLLM, the OpenAI Python SDK's built-in retry, most custom middleware — was treating these two conditions identically.

What the split looks like now

429 with slow_down code — your request rate is rising too quickly. The right response is to back off and hold rate for a meaningful interval before resuming. If the Retry-After header is present, wait at least that long. If it isn't, apply exponential backoff. Do not resume at your previous send rate immediately after the wait — ramp back up gradually.

503 with server_is_overloaded code — OpenAI's inference infrastructure is temporarily saturated. This is transient and typically resolves within seconds to a few minutes. The right response is short exponential backoff, with the expectation that you can return to your full request rate once the retry succeeds. This is explicitly not your traffic being too fast.

Both responses can include a Retry-After header. When it's present, it's a hard floor — don't retry sooner. When it's absent, use your backoff curve.

What breaks in existing retry code

The problem is that most gateway retry implementations key off HTTP status code alone. They see a 429 and apply backoff. They see a 5xx and either fail fast or apply the same backoff. The new 503 server_is_overloaded case requires a retry with a reasonably short wait — not a fast fail and not a prolonged throttle.

Here's the mental model that now has to live in your retry layer:

If your client is currently treating all 429s as "throttle me" and all 503s as "fail or retry with the same logic as 429", you have both false positives and false negatives after this change.

How Anthropic and Google handle the same problem — and where OpenAI's split lands

For context: Anthropic uses a 529 status for model overload, completely separating it from the 429 rate-limit space at the HTTP level. This is cleaner — your HTTP middleware can distinguish the two without parsing the body. The tradeoff is that 529 is a non-standard status code not in the RFC, which surprises some gateway software.

Google's Gemini API returns a 429 ResourceExhausted for both conditions (quota and capacity), but includes a structured @type: google.rpc.ErrorInfo body with a reason field (RATE_LIMIT_EXCEEDED vs CONCURRENCY_LIMIT_EXCEEDED) that lets you distinguish them — but only if you parse the gRPC-style error body, which most REST clients don't.

DeepSeek V4 introduced peak/off-peak pricing earlier this year and issues 429s with a too_many_requests code for both quota and capacity limits, with no body-level distinction.

OpenAI's new approach — keeping the 429 for rate limit, promoting model saturation to a 503 with a named code in the body — is a middle ground. It's more correct than treating everything as 429, and more standard than Anthropic's 529, but it still requires body parsing to distinguish server_is_overloaded from a genuine 503 service failure.

The routing layer consequence

If you're using TheRouter or any AI gateway to route traffic across providers, this changes when you should count a response as a signal to shift load. A slow_down 429 from OpenAI means you should slow down requests to OpenAI. A server_is_overloaded 503 means OpenAI has a transient capacity issue — a valid signal to redirect that request to a fallback provider if your latency budget can't absorb a short wait.

Previously, both conditions looked identical in your gateway error log, and you were forced to pick one interpretation. Now you can route them differently:

• slow_down → honor rate limit; keep routing to OpenAI but at reduced rate
• server_is_overloaded → optionally fail-over to a backup provider or queue for immediate retry; do not reduce your rate limit on the OpenAI lane

What to audit in your client and gateway now

1. Check your retry library's status code handling. If it fails fast on any 503, it will now fast-fail on transient OpenAI capacity spikes instead of retrying.

2. Check whether you parse the error body at all. If your code checks response.status === 503 and raises an exception without reading error.code, you'll never see server_is_overloaded.

3. If you gate on error body codes, add server_is_overloaded as a retryable condition alongside existing 429 retry logic.

4. Log the code alongside the status. Aggregating "429 rate limit" vs "429 slow_down" vs "503 server_is_overloaded" in your observability layer will give you a much cleaner signal on what's actually happening during high-load periods.

The OpenAI API changelog lists this as a minor update. In practice, it's the first time these two conditions have been observable at the API level, and getting your retry logic right matters most when you're already under load.