LLM Batch API у разных providers: production guide для async AI workloads

30-second answer: batch APIs подходят, когда LLM job не требует немедленного ответа. OpenAI, Anthropic, DashScope и Gemini-style batch paths могут снизить cost и поднять throughput, но меняют reliability model: нужно upload work, create async job, poll status, download output/error files и reconcile каждую строку. Используйте TheRouter для real-time OpenAI-compatible routing paths, которые проверены сегодня; provider batch APIs рассматривайте как соседние async lanes, пока нет live evidence для end-to-end gateway path.

This guide fills the gap between our LLM API cost optimization guide and the LLM API error-handling reference. It is about batch mechanics, failure boundaries, and orchestration patterns — not another generic “save money with cheaper models” post.

OpenAI-совместимость означает, что провайдер предоставляет endpoint chat-completions, чей контракт запроса и ответа достаточно близок к API OpenAI, чтобы немодифицированный вызов OpenAI SDK работал после замены трёх значений: API key, base URL, название модели. Минимальная поверхность на практике —POST /v1/chat/completions с messages, model и потоковым ответом в форме OpenAI.

Когда batch — правильный execution mode

Use batch when the business can wait and the unit of work can be reconciled later. Good candidates include offline evaluations, dataset classification, document enrichment, embeddings, moderation backfills, synthetic test generation, and nightly quality reports. OpenAI describes Batch as asynchronous processing for jobs that do not need immediate responses, with a 24-hour turnaround target, 50% lower cost, and a separate higher-limit pool. Anthropic documents Message Batches for large volumes of Messages requests, also at 50% of standard API prices, with most batches finishing faster but expiring after 24 hours if unfinished. DashScope Model Studio documents an OpenAI-compatible Batch File API at 50% of real-time calls for large-scale workloads where latency is not critical.

Do not use batch for live chat, interactive agent loops, customer-visible tool calls, payment decisions, or anything where a late answer is equivalent to a failed answer. Batch reduces unit economics; it does not remove product latency requirements.

Provider comparison table

The useful pattern is not “pick one winner.” It is to split offline work by provider capability, model quality, region, quota, and reconciliation cost.

Механика OpenAI Batch API

OpenAI's flow is file-first. You write JSONL rows with a unique custom_id, upload the file with purpose batch, create a batch for an endpoint such as /v1/chat/completions, /v1/responses, /v1/embeddings, or supported media endpoints, poll the batch object, then download result files through the Files API. Statuses include validating, in_progress, finalizing, completed, failed, expired, cancelling, and cancelled.

A minimal row looks like this:

{"custom_id":"eval-001","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5","messages":[{"role":"user","content":"Classify this support ticket."}],"max_tokens":200}}

The custom_id is not decoration. It is the join key that lets you merge responses back into your database and distinguish successful, failed, expired, and retried rows. If you cannot design a stable key, you are not ready to batch the workload.

Механика Anthropic Message Batches

Anthropic uses a Messages-native request list rather than OpenAI-style endpoint rows. Each request has a custom_id and params, and the provider processes the requests independently. The docs call out a 100,000-message or 256 MB batch limit, a 24-hour expiry window, and 29-day result availability. They also state that active Claude models support Message Batches, while specific synchronous-only parameters are rejected.

The architectural implication is simple: make your batch runner parameter-aware. A job that streams in production cannot be copied into batch by flipping one flag. Remove streaming, confirm max_tokens, and validate unsupported parameters before submitting the whole file.

Механика DashScope batch

DashScope's OpenAI-compatible Batch File API is especially relevant for teams already using Qwen through compatible-mode/v1. The documented flow mirrors OpenAI SDK concepts: upload a JSONL file, create a batch with an endpoint such as /v1/chat/completions, poll status, and download output and error files. The docs also distinguish Beijing and Singapore endpoints, including workspace-specific Singapore domains.

That region detail matters. A batch route that works in Beijing may not support the same model set in Singapore, and the docs explicitly list different supported models by region. For Qwen 3.7, 3.6, and 3.5 hybrid-thinking models, also account for thinking-mode defaults and thinking-token billing.

Gateway orchestration pattern

The truthful TheRouter claim is narrow and useful: TheRouter routes OpenAI-compatible real-time requests through configured providers and supports provider/model routing and fallback when the live product path supports it. For batch, frame the implementation as orchestration around provider batch lanes unless you have verified a native TheRouter batch path.

A practical architecture is:

1. app tags jobs as realtime or offline;
2. realtime traffic goes through TheRouter's OpenAI-compatible route and provider pages such as OpenAI, Anthropic, DashScope, and Google;
3. offline traffic goes to a batch runner that chooses the provider lane;
4. batch results are reconciled back into the same usage ledger and observability model;
5. failures feed the retry and fallback runbook, not an infinite retry loop.

Link model-level decisions to current catalog pages such as GPT-5, Claude Sonnet 4.6, and Qwen3.7 Plus, but verify batch support in provider docs before assuming a model can run asynchronously.

Failure handling и reconciliation

Batch failures are row-level and job-level. Job-level failures include invalid files, unsupported parameters, regional model mismatch, expired jobs, and cancellation. Row-level failures include content-policy errors, token-limit errors, malformed request bodies, and provider-specific model errors.

Your runner should store:

• batch_provider, batch_id, input_file_id, output_file_id, and error_file_id;
• custom_id, source record id, prompt version, model id, and expected schema;
• job state transitions with timestamps;
• row-level success, provider error, retry decision, and final disposition;
• cost estimate before submission and measured usage after completion.

For retry rules, reuse the discipline from the timeout, retries, and idempotency guide candidate and the fallback routing guide: retry only safe failures, cap retries, and never duplicate side-effectful actions.

Production checklist

1. Меняйте три значения, а не три SDK. Замените api_key, base_url и model в существующем клиенте OpenAI. Код запроса и ответа оставьте неизменным.
2. Сопоставьте model ID явно. ID модели у целевого провайдера почти никогда не совпадает с OpenAI ID. Держите словарь { openai_id: target_id } вне бизнес-логики.
3. Проверьте формат streaming. SSE-чанки должны соответствовать контракту OpenAI: data: {...} + data: [DONE]. Прогоните один streaming вызов до продакшена.
4. Проверьте rate-limit заголовки. Некоторые провайдеры не возвращают x-ratelimit-*. Добавьте обёртку с безопасным дефолтом.
5. Оставьте путь отката. Раскатайте замену под feature flag, гоняйте оба endpoint в shadow-режиме 24 часа, затем переключайтесь.

Before moving a workload to batch, confirm:

• the user experience can tolerate asynchronous completion;
• every input row has a stable custom_id and source record id;
• provider docs confirm the target endpoint, model, region, and parameters;
• streaming and interactive tool loops are removed from batch jobs;
• output and error files are downloaded before retention windows expire;
• expired jobs are treated as partial completion, not silent success;
• finance dashboards separate real-time and batch usage;
• security review covers uploaded file contents and retention rules.

FAQ

Batch API — это просто более дешевый chat completion?

No. It is a different execution mode: async submission, later retrieval, row reconciliation, different expiry semantics, and sometimes different unsupported parameters.

Может ли TheRouter автоматически batch все providers?

Do not make that claim. The verified claim is OpenAI-compatible routing for configured providers, with fallback where live paths support it. Use provider batch APIs directly or through a separately verified batch runner until a native gateway batch path is proven.

Что переносить первым?

Start with evaluation sets, offline classification, embeddings, and document enrichment. Avoid live agents, user-visible chat, and workflows with irreversible side effects.

Sources

• OpenAI Batch API docs — https://developers.openai.com/api/docs/guides/batch — retrieved 2026-08-01.
• Anthropic Batch processing docs — https://platform.claude.com/docs/en/build-with-claude/batch-processing — retrieved 2026-08-01.
• Alibaba Cloud Model Studio OpenAI-compatible Batch File API — https://www.alibabacloud.com/help/en/model-studio/batch-interfaces-compatible-with-openai/ — retrieved 2026-08-01.
• Google Gemini API pricing — https://ai.google.dev/gemini-api/docs/pricing — retrieved via search snippet 2026-08-01 because the page redirected during fetch.
• TheRouter quickstart — https://therouter.ai/docs/quickstart/ — retrieved 2026-08-01.
• TheRouter fallback routing guide — https://therouter.ai/docs/guides/routing/model-fallbacks/ — retrieved 2026-08-01.