TheRouter.aiTheRouter.ai
← 全部文章

跨 Provider 的 LLM API 超时、重试与幂等：生产可靠性指南

面向 OpenAI-compatible LLM API 的实战指南：如何设置超时预算、重试规则、幂等键、防重复输出和 fallback 决策。

· TheRouter

30 秒答案：安全的 LLM API 重试，第一步是判断上一次 attempt 是明确失败、明确成功，还是可能仍在生成。 Retry 429, connection, timeout, and 5xx failures only inside a bounded budget; honor Retry-After when a provider sends it; never retry side-effecting tool calls unless the output sink is idempotent; and use fallback when another provider/model route is safer than waiting on the same route. This guide covers the implementation pattern we use for OpenAI-compatible systems without promising zero downtime or exactly-once model execution.

OpenAI documents Retry-After guidance for 429 responses and brief retries for 500/503 errors. Anthropic official search snippets describe SDK retries for connection errors, rate limits, and 5xx errors. DeepSeek recommends pacing 429 traffic and retrying 500/503 after a brief wait. DashScope documents OpenAI-compatible endpoints plus provider-specific 429 and parameter errors.

OpenAI 兼容指供应商提供一个 chat-completions 接口，其请求与响应结构与 OpenAI API 契约足够接近——只需替换三个值（API key、base URL、模型名），原来的 OpenAI SDK 调用即可直接工作。最小实践面是POST /v1/chat/completions 带 messages、model， 并返回 OpenAI 形式的流式响应。

为什么 LLM API 的重试更难

A normal JSON API often gives you a clean success or failure. LLM APIs add long-running generation, streaming partial output, and tool or workflow side effects. A client timeout can fire while the provider is still producing tokens; your UI may already have shown part of the answer; and a retry can duplicate an email, ticket update, database write, or other action.

That is why the retry question is not simply "is this HTTP status retryable?" The real question is: can another attempt produce duplicate user-visible work? If the answer is yes, make the sink idempotent before adding automatic retries.

失败状态表

Failure stateTypical signalSafe first actionRetry?Fallback?
Request rejected before execution400, 401, 403, invalid parametersFix request/auth/configNoNo
Quota or billing exhausted402, credit exhausted, spend limitAdd quota or change accountNoMaybe, if policy allows another provider
Rate limited429, Retry-After, QPS/QPM messageWait, reduce concurrencyYes, after delayYes, if user latency budget is short
Provider transient error500, 503, overloadedBrief backoff with jitterYesYes, after small retry budget
Local network/connect errorconnection reset, DNS/TLS timeoutRetry same route once or twiceYesYes, if repeated
Client read timeoutno final response before deadlineCheck whether output reached sinkCarefullyPrefer recoverable error or deduped fallback
Streaming interruptionpartial SSE chunks, no final stopMark partial output incompleteRarely resume; usually restart with dedupeMaybe
Tool side effect uncertainfunction/tool call may have runQuery sink by idempotency keyOnly after dedupe checkNo until state is known

This table deliberately separates configuration errors from transient failures. Retrying an invalid API key or malformed payload only burns quota and hides the actual fix.

超时预算：不要只设一个 timeout

Set separate clocks for connect timeout, first-token timeout, stream-idle timeout, and the end-to-end deadline. A support-chat workload might allow 2 seconds to connect, 20 seconds to first token, 15 seconds of stream idle, and 45 seconds total. Offline enrichment can wait longer, but the retry count and token budget still need hard limits.

A single global timeout usually fails both sides: it is too short for reasoning models and too long for simple classification. Budget by workload, not by SDK default.

Provider 表面的重试策略

Provider surfaceRetryable signals we rely onNon-retryable signalsNotes
OpenAI429 with Retry-After, 500, 503401/403 auth, billing/spend/quota errors, invalid requestOpenAI says to pace 429 requests and retry 500/503 after a brief wait.
Anthropicconnection errors, 429, 5xx400/401/403 request or auth failuresOfficial snippets state SDKs retry transient failures with exponential backoff; rate-limit docs mention retry-after.
DashScope / Alibaba Cloud429 QPS/QPM, occasional service-side 5xxinvalid model parameters, unsupported stream mode, bad bodyDashScope documents OpenAI-compatible base_url setup and streaming examples; many parameter errors should be fixed, not retried.
DeepSeek429, 500, 503400/401/402/422DeepSeek recommends pacing 429 traffic and retrying 500/503 after a brief wait.
OpenAI-compatible gatewayupstream timeout, upstream 429/5xx, route failuremalformed OpenAI payload, policy-blocked requestGateway fallback helps only when configured and supported by the live path; it is not a zero-downtime guarantee.

Use this table as a starting point, then tune by workload. A coding-agent request with tool calls should be more conservative than a stateless classification request.

真正能防重复的幂等模式

The LLM provider may not know whether your business action is safe to repeat. Your app must know. Use three layers: a client request ID, an output sink dedupe key, and a tool-call execution guard. Store generated artefacts under a stable (workflow_id, step_id, attempt_group) key so retry updates or supersedes the same record instead of creating a second record.

type RetryState = {
  requestId: string;
  attempt: number;
  deadlineMs: number;
  outputKey: string;
};

function shouldRetry(status: number, attempt: number) {
  if (attempt >= 2) return false;
  if (status === 429) return true;
  if (status >= 500 && status < 600) return true;
  return false;
}

For tool calls, check (request_id, tool_name, tool_args_hash) before execution. If the call already ran, return the stored result instead of executing the side effect again.

Retry、fallback，还是 fail fast？

Use this decision tree: fail fast for malformed, unauthorized, or over-quota requests; honor Retry-After when the provider asks you to wait; retry stateless workloads once or twice with exponential backoff and jitter; mark partial streaming output incomplete instead of appending a second answer; fallback only when the workload accepts a different model/provider; and stop when the total deadline is nearly spent.

With TheRouter, fallback chains can be configured by passing a models array and, where applicable, provider routing options. The docs say TheRouter attempts model candidates in priority order and reports the model that ultimately served the response. Treat that as a reliability control, not a promise that every provider failure can be hidden.

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.therouter.ai/v1",
  apiKey: process.env.THEROUTER_API_KEY,
  maxRetries: 0,
  timeout: 45_000,
});

const completion = await client.chat.completions.create({
  model: "anthropic/claude-sonnet-4.5",
  extra_body: { models: ["openai/gpt-5-mini", "google/gemini-3-flash-preview"] },
  messages: [{ role: "user", content: "Classify this support ticket." }],
});

Streaming 重试：默认 partial output 不可 resume

SSE streams are excellent for latency, but retry semantics are stricter. If a stream drops after 400 tokens, most chat-completion paths cannot resume from token 401 with a provider guarantee. Restarting the call can generate a different answer.

Do not stream directly into a permanent record. Buffer, then commit at a finish signal. If you must display live tokens, tag the visible answer as incomplete until the final chunk arrives. If the stream fails, either ask the user to retry or start a new attempt that replaces the previous draft; never concatenate both outputs. For protocol details, see our SSE streaming guide.

观测字段：调优前先记录

FieldWhy it matters
request_idJoins app logs, gateway logs, provider errors, and output records
attempt and attempt_groupDistinguishes retry attempts from separate user requests
provider, model, response.modelShows which route actually served the request
status, error.type, error.codeSeparates config failures from transient failures
retry_after_msProves whether the client honored provider pacing
deadline_remaining_msExplains why fallback did or did not run
input_tokens, output_tokens, cached_tokensConnects reliability events to cost per accepted outcome
partial_stream_committedPrevents duplicate or concatenated streaming output

生产 checklist

  • Set maxRetries consciously; do not stack SDK retries, gateway retries, and job-runner retries without one owner.
  • Keep one end-to-end deadline across all attempts.
  • Honor provider Retry-After headers where they exist.
  • Use jittered exponential backoff for transient failures.
  • Do not retry 400/401/403/402-style failures.
  • Add idempotency keys to output sinks and tool execution.
  • Buffer streaming output before permanent writes.
  • Cap retries by token budget and user-visible latency.
  • Fall back only to models/providers that are acceptable for the workload.
  • Alert on retry storms, fallback rate spikes, and duplicate-output guard hits.

FAQ

Should we retry every 429?

No. Retry only if the request is still useful after the delay. If the deadline is short, use a configured fallback route or return a recoverable error instead.

Should the SDK or our app own retries?

For prototypes, SDK defaults are fine. In production, we prefer one workflow-level retry owner because it can see user deadlines, idempotency keys, fallback policy, and output sinks.

Can fallback replace retries?

No. Fallback is a route decision; retry is an attempt decision. Use a small same-route retry for transient errors, then fallback if the workload accepts another provider/model. See our fallback routing guide.

How do we avoid duplicate tool calls?

Execute tools through a durable guard keyed by request ID, tool name, and normalized arguments. If the same call appears again after a retry, return the stored result instead of executing the side effect twice.

What changes for OpenAI-compatible APIs?

The request shape is portable, but error semantics are not identical. A single /v1/chat/completions client can call OpenAI, DashScope, DeepSeek, or TheRouter, but you still need provider-aware retry rules and citations for each behavior.

Sources

客服支持