TheRouter.aiTheRouter.ai
← 全部文章

跨 Provider 的 LLM Batch API：异步 AI 工作负载生产指南

一份跨 provider 的 LLM Batch API 指南：什么时候使用异步处理，OpenAI、Anthropic、DashScope 与 Gemini 有何差异，以及如何在不夸大 gateway 原生 batch 能力的前提下编排批量任务。

· TheRouter

30 秒答案： 当 LLM 任务不需要立即返回结果时，batch API 才是正确路径。OpenAI、Anthropic、DashScope 和 Gemini 风格的 batch 路径可以降低成本、提高吞吐，但也会改变可靠性模型：你需要上传任务、创建异步 job、轮询状态、下载 output/error 文件，再逐行对账。TheRouter 适合今天已经验证的实时 OpenAI-compatible 路由；除非你有端到端 live evidence，否则应把 provider batch API 当作相邻的异步通道。

这篇文章补上 LLM API 成本优化指南 和 LLM API 错误处理参考 之间的空白。重点是 batch 机制、失败边界和编排模式，而不是又一篇“用更便宜模型省钱”的泛泛文章。

OpenAI 兼容指供应商提供一个 chat-completions 接口，其请求与响应结构与 OpenAI API 契约足够接近——只需替换三个值（API key、base URL、模型名），原来的 OpenAI SDK 调用即可直接工作。最小实践面是POST /v1/chat/completions 带 messages、model， 并返回 OpenAI 形式的流式响应。

什么时候 batch 是正确执行模式

Use batch when the business can wait and the unit of work can be reconciled later. Good candidates include offline evaluations, dataset classification, document enrichment, embeddings, moderation backfills, synthetic test generation, and nightly quality reports. OpenAI describes Batch as asynchronous processing for jobs that do not need immediate responses, with a 24-hour turnaround target, 50% lower cost, and a separate higher-limit pool. Anthropic documents Message Batches for large volumes of Messages requests, also at 50% of standard API prices, with most batches finishing faster but expiring after 24 hours if unfinished. DashScope Model Studio documents an OpenAI-compatible Batch File API at 50% of real-time calls for large-scale workloads where latency is not critical.

Do not use batch for live chat, interactive agent loops, customer-visible tool calls, payment decisions, or anything where a late answer is equivalent to a failed answer. Batch reduces unit economics; it does not remove product latency requirements.

Provider 对比表

ProviderBatch shapeCompletion / expiryCost leverMain caveat
OpenAIUpload JSONL with custom_id, create /v1/batches, retrieve output and error filescompletion_window: "24h"; expired if not completed in the window50% discount versus synchronous APIs; separate batch rate limitsOne input file targets one endpoint and one model; streaming is not the result shape
AnthropicCreate messages/batches with request objects and custom_id valuesresults when all messages finish or after 24h; results downloadable for 29 days50% of standard API pricesSome synchronous parameters, including stream: true, Fast mode, and max_tokens: 0, are not supported
DashScope / Model StudioOpenAI-compatible file input API through regional compatible-mode/v1 endpointsAPI uses completion_window: "24h" and result/error file retrieval50% of real-time callsSupported model list differs by region; Qwen thinking mode can add thinking-token cost
GeminiBatch mode is advertised as a 50% cost-reduction option in official pricing snippetsverify current docs before launch50% discount on paid models per official pricing snippetsGoogle docs redirected during retrieval; confirm model, region, and client compatibility in the live console

The useful pattern is not “pick one winner.” It is to split offline work by provider capability, model quality, region, quota, and reconciliation cost.

OpenAI Batch API 机制

OpenAI's flow is file-first. You write JSONL rows with a unique custom_id, upload the file with purpose batch, create a batch for an endpoint such as /v1/chat/completions, /v1/responses, /v1/embeddings, or supported media endpoints, poll the batch object, then download result files through the Files API. Statuses include validating, in_progress, finalizing, completed, failed, expired, cancelling, and cancelled.

A minimal row looks like this:

{"custom_id":"eval-001","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5","messages":[{"role":"user","content":"Classify this support ticket."}],"max_tokens":200}}

The custom_id is not decoration. It is the join key that lets you merge responses back into your database and distinguish successful, failed, expired, and retried rows. If you cannot design a stable key, you are not ready to batch the workload.

Anthropic Message Batches 机制

Anthropic uses a Messages-native request list rather than OpenAI-style endpoint rows. Each request has a custom_id and params, and the provider processes the requests independently. The docs call out a 100,000-message or 256 MB batch limit, a 24-hour expiry window, and 29-day result availability. They also state that active Claude models support Message Batches, while specific synchronous-only parameters are rejected.

The architectural implication is simple: make your batch runner parameter-aware. A job that streams in production cannot be copied into batch by flipping one flag. Remove streaming, confirm max_tokens, and validate unsupported parameters before submitting the whole file.

DashScope batch 机制

DashScope's OpenAI-compatible Batch File API is especially relevant for teams already using Qwen through compatible-mode/v1. The documented flow mirrors OpenAI SDK concepts: upload a JSONL file, create a batch with an endpoint such as /v1/chat/completions, poll status, and download output and error files. The docs also distinguish Beijing and Singapore endpoints, including workspace-specific Singapore domains.

That region detail matters. A batch route that works in Beijing may not support the same model set in Singapore, and the docs explicitly list different supported models by region. For Qwen 3.7, 3.6, and 3.5 hybrid-thinking models, also account for thinking-mode defaults and thinking-token billing.

Gateway 编排模式

The truthful TheRouter claim is narrow and useful: TheRouter routes OpenAI-compatible real-time requests through configured providers and supports provider/model routing and fallback when the live product path supports it. For batch, frame the implementation as orchestration around provider batch lanes unless you have verified a native TheRouter batch path.

A practical architecture is:

  1. app tags jobs as realtime or offline;
  2. realtime traffic goes through TheRouter's OpenAI-compatible route and provider pages such as OpenAI, Anthropic, DashScope, and Google;
  3. offline traffic goes to a batch runner that chooses the provider lane;
  4. batch results are reconciled back into the same usage ledger and observability model;
  5. failures feed the retry and fallback runbook, not an infinite retry loop.

Link model-level decisions to current catalog pages such as GPT-5, Claude Sonnet 4.6, and Qwen3.7 Plus, but verify batch support in provider docs before assuming a model can run asynchronously.

故障处理与结果对账

Batch failures are row-level and job-level. Job-level failures include invalid files, unsupported parameters, regional model mismatch, expired jobs, and cancellation. Row-level failures include content-policy errors, token-limit errors, malformed request bodies, and provider-specific model errors.

Your runner should store:

  • batch_provider, batch_id, input_file_id, output_file_id, and error_file_id;
  • custom_id, source record id, prompt version, model id, and expected schema;
  • job state transitions with timestamps;
  • row-level success, provider error, retry decision, and final disposition;
  • cost estimate before submission and measured usage after completion.

For retry rules, reuse the discipline from the timeout, retries, and idempotency guide candidate and the fallback routing guide: retry only safe failures, cap retries, and never duplicate side-effectful actions.

生产检查清单

  1. 替换三个值，不是三个 SDK。在现有 OpenAI 客户端里改 api_key、base_url、model。请求与响应代码保持不变。
  2. 显式映射 model ID。目标供应商的 model id 几乎不会和 OpenAI 完全一致。 在业务代码之外维护一份 { openai_id: target_id } 映射。
  3. 验证流式格式。SSE 分片必须遵循 OpenAI 的 data: {...} + data: [DONE] 契约。切生产前先跑一次流式调用。
  4. 检查限流响应头。部分供应商不返回 x-ratelimit-*。 在包装层 做缺省兜底，缺头不要崩。
  5. 留回滚路径。用 feature flag 切流；新旧 endpoint 影子并行 24 小时，再正式切换。

Before moving a workload to batch, confirm:

  • the user experience can tolerate asynchronous completion;
  • every input row has a stable custom_id and source record id;
  • provider docs confirm the target endpoint, model, region, and parameters;
  • streaming and interactive tool loops are removed from batch jobs;
  • output and error files are downloaded before retention windows expire;
  • expired jobs are treated as partial completion, not silent success;
  • finance dashboards separate real-time and batch usage;
  • security review covers uploaded file contents and retention rules.

FAQ

Batch API 只是更便宜的 chat completion 吗？

No. It is a different execution mode: async submission, later retrieval, row reconciliation, different expiry semantics, and sometimes different unsupported parameters.

TheRouter 能自动 batch 所有 provider 吗？

Do not make that claim. The verified claim is OpenAI-compatible routing for configured providers, with fallback where live paths support it. Use provider batch APIs directly or through a separately verified batch runner until a native gateway batch path is proven.

应该先迁移什么？

Start with evaluation sets, offline classification, embeddings, and document enrichment. Avoid live agents, user-visible chat, and workflows with irreversible side effects.

Sources

本文涉及的模型

客服支持