TheRouter.aiTheRouter.ai
← All articles

LLM API Timeouts, Retries, and Idempotency Across Providers: A Production Reliability Guide

A practical guide to timeout budgets, retry rules, idempotency keys, duplicate-output prevention, and fallback decisions for OpenAI-compatible LLM APIs.

· TheRouter

A 30-second answer: safe LLM API retries start by deciding whether the previous attempt definitely failed, definitely succeeded, or may still be completing. Retry 429, connection, timeout, and 5xx failures only inside a bounded budget; honor Retry-After when a provider sends it; never retry side-effecting tool calls unless the output sink is idempotent; and use fallback when another provider/model route is safer than waiting on the same route. This guide covers the implementation pattern we use for OpenAI-compatible systems without promising zero downtime or exactly-once model execution.

OpenAI documents Retry-After guidance for 429 responses and brief retries for 500/503 errors. Anthropic official search snippets describe SDK retries for connection errors, rate limits, and 5xx errors. DeepSeek recommends pacing 429 traffic and retrying 500/503 after a brief wait. DashScope documents OpenAI-compatible endpoints plus provider-specific 429 and parameter errors.

OpenAI-compatible means a provider exposes a chat-completions endpoint whose request and response shape matches the OpenAI API contract closely enough that an unmodified OpenAI SDK call works against it after swapping three values: API key, base URL, and model name. The minimum surface in practice is POST /v1/chat/completions with messages, model, and an OpenAI-shaped streaming response.

Why retry logic is harder for LLM APIs

A normal JSON API often gives you a clean success or failure. LLM APIs add long-running generation, streaming partial output, and tool or workflow side effects. A client timeout can fire while the provider is still producing tokens; your UI may already have shown part of the answer; and a retry can duplicate an email, ticket update, database write, or other action.

That is why the retry question is not simply "is this HTTP status retryable?" The real question is: can another attempt produce duplicate user-visible work? If the answer is yes, make the sink idempotent before adding automatic retries.

Failure-state table

Failure stateTypical signalSafe first actionRetry?Fallback?
Request rejected before execution400, 401, 403, invalid parametersFix request/auth/configNoNo
Quota or billing exhausted402, credit exhausted, spend limitAdd quota or change accountNoMaybe, if policy allows another provider
Rate limited429, Retry-After, QPS/QPM messageWait, reduce concurrencyYes, after delayYes, if user latency budget is short
Provider transient error500, 503, overloadedBrief backoff with jitterYesYes, after small retry budget
Local network/connect errorconnection reset, DNS/TLS timeoutRetry same route once or twiceYesYes, if repeated
Client read timeoutno final response before deadlineCheck whether output reached sinkCarefullyPrefer recoverable error or deduped fallback
Streaming interruptionpartial SSE chunks, no final stopMark partial output incompleteRarely resume; usually restart with dedupeMaybe
Tool side effect uncertainfunction/tool call may have runQuery sink by idempotency keyOnly after dedupe checkNo until state is known

This table deliberately separates configuration errors from transient failures. Retrying an invalid API key or malformed payload only burns quota and hides the actual fix.

Timeout budgets: four clocks, not one number

Set separate clocks for connect timeout, first-token timeout, stream-idle timeout, and the end-to-end deadline. A support-chat workload might allow 2 seconds to connect, 20 seconds to first token, 15 seconds of stream idle, and 45 seconds total. Offline enrichment can wait longer, but the retry count and token budget still need hard limits.

A single global timeout usually fails both sides: it is too short for reasoning models and too long for simple classification. Budget by workload, not by SDK default.

Retry policy by provider surface

Provider surfaceRetryable signals we rely onNon-retryable signalsNotes
OpenAI429 with Retry-After, 500, 503401/403 auth, billing/spend/quota errors, invalid requestOpenAI says to pace 429 requests and retry 500/503 after a brief wait.
Anthropicconnection errors, 429, 5xx400/401/403 request or auth failuresOfficial snippets state SDKs retry transient failures with exponential backoff; rate-limit docs mention retry-after.
DashScope / Alibaba Cloud429 QPS/QPM, occasional service-side 5xxinvalid model parameters, unsupported stream mode, bad bodyDashScope documents OpenAI-compatible base_url setup and streaming examples; many parameter errors should be fixed, not retried.
DeepSeek429, 500, 503400/401/402/422DeepSeek recommends pacing 429 traffic and retrying 500/503 after a brief wait.
OpenAI-compatible gatewayupstream timeout, upstream 429/5xx, route failuremalformed OpenAI payload, policy-blocked requestGateway fallback helps only when configured and supported by the live path; it is not a zero-downtime guarantee.

Use this table as a starting point, then tune by workload. A coding-agent request with tool calls should be more conservative than a stateless classification request.

Idempotency patterns that actually prevent duplicates

The LLM provider may not know whether your business action is safe to repeat. Your app must know. Use three layers: a client request ID, an output sink dedupe key, and a tool-call execution guard. Store generated artefacts under a stable (workflow_id, step_id, attempt_group) key so retry updates or supersedes the same record instead of creating a second record.

type RetryState = {
  requestId: string;
  attempt: number;
  deadlineMs: number;
  outputKey: string;
};

function shouldRetry(status: number, attempt: number) {
  if (attempt >= 2) return false;
  if (status === 429) return true;
  if (status >= 500 && status < 600) return true;
  return false;
}

For tool calls, check (request_id, tool_name, tool_args_hash) before execution. If the call already ran, return the stored result instead of executing the side effect again.

Retry, fallback, or fail fast?

Use this decision tree: fail fast for malformed, unauthorized, or over-quota requests; honor Retry-After when the provider asks you to wait; retry stateless workloads once or twice with exponential backoff and jitter; mark partial streaming output incomplete instead of appending a second answer; fallback only when the workload accepts a different model/provider; and stop when the total deadline is nearly spent.

With TheRouter, fallback chains can be configured by passing a models array and, where applicable, provider routing options. The docs say TheRouter attempts model candidates in priority order and reports the model that ultimately served the response. Treat that as a reliability control, not a promise that every provider failure can be hidden.

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.therouter.ai/v1",
  apiKey: process.env.THEROUTER_API_KEY,
  maxRetries: 0,
  timeout: 45_000,
});

const completion = await client.chat.completions.create({
  model: "anthropic/claude-sonnet-4.5",
  extra_body: { models: ["openai/gpt-5-mini", "google/gemini-3-flash-preview"] },
  messages: [{ role: "user", content: "Classify this support ticket." }],
});

Streaming retries: assume partial output is not resumable

SSE streams are excellent for latency, but retry semantics are stricter. If a stream drops after 400 tokens, most chat-completion paths cannot resume from token 401 with a provider guarantee. Restarting the call can generate a different answer.

Do not stream directly into a permanent record. Buffer, then commit at a finish signal. If you must display live tokens, tag the visible answer as incomplete until the final chunk arrives. If the stream fails, either ask the user to retry or start a new attempt that replaces the previous draft; never concatenate both outputs. For protocol details, see our SSE streaming guide.

Observability: the fields you need before tuning

FieldWhy it matters
request_idJoins app logs, gateway logs, provider errors, and output records
attempt and attempt_groupDistinguishes retry attempts from separate user requests
provider, model, response.modelShows which route actually served the request
status, error.type, error.codeSeparates config failures from transient failures
retry_after_msProves whether the client honored provider pacing
deadline_remaining_msExplains why fallback did or did not run
input_tokens, output_tokens, cached_tokensConnects reliability events to cost per accepted outcome
partial_stream_committedPrevents duplicate or concatenated streaming output

Production checklist

  • Set maxRetries consciously; do not stack SDK retries, gateway retries, and job-runner retries without one owner.
  • Keep one end-to-end deadline across all attempts.
  • Honor provider Retry-After headers where they exist.
  • Use jittered exponential backoff for transient failures.
  • Do not retry 400/401/403/402-style failures.
  • Add idempotency keys to output sinks and tool execution.
  • Buffer streaming output before permanent writes.
  • Cap retries by token budget and user-visible latency.
  • Fall back only to models/providers that are acceptable for the workload.
  • Alert on retry storms, fallback rate spikes, and duplicate-output guard hits.

FAQ

Should we retry every 429?

No. Retry only if the request is still useful after the delay. If the deadline is short, use a configured fallback route or return a recoverable error instead.

Should the SDK or our app own retries?

For prototypes, SDK defaults are fine. In production, we prefer one workflow-level retry owner because it can see user deadlines, idempotency keys, fallback policy, and output sinks.

Can fallback replace retries?

No. Fallback is a route decision; retry is an attempt decision. Use a small same-route retry for transient errors, then fallback if the workload accepts another provider/model. See our fallback routing guide.

How do we avoid duplicate tool calls?

Execute tools through a durable guard keyed by request ID, tool name, and normalized arguments. If the same call appears again after a retry, return the stored result instead of executing the side effect twice.

What changes for OpenAI-compatible APIs?

The request shape is portable, but error semantics are not identical. A single /v1/chat/completions client can call OpenAI, DashScope, DeepSeek, or TheRouter, but you still need provider-aware retry rules and citations for each behavior.

Sources

Customer Support