LLM API Streaming Across Providers: SSE Quirks, Token Buffering, and What Breaks When You Switch

Every major LLM API supports streaming via Server-Sent Events (SSE). You set stream: true, and tokens arrive incrementally instead of in one final response. Simple in theory. In practice, OpenAI, Anthropic, DashScope, and DeepSeek each implement SSE differently enough that switching providers — or routing through a gateway — breaks your streaming client in ways that are not obvious until production.

This guide documents the concrete differences. We tested every provider's streaming output and recorded what the wire protocol actually looks like: event types, chunk shapes, termination signals, error behavior mid-stream, and tool-call serialization. If you run a multi-provider setup or are considering one, this is the reference we wish existed when we built our own streaming normalization layer.

OpenAI-compatible means a provider exposes a chat-completions endpoint whose request and response shape matches the OpenAI API contract closely enough that an unmodified OpenAI SDK call works against it after swapping three values: API key, base URL, and model name. The minimum surface in practice is POST /v1/chat/completions with messages, model, and an OpenAI-shaped streaming response.

What SSE streaming actually looks like

All four providers return Content-Type: text/event-stream and send newline-delimited chunks. That is where the similarity ends.

OpenAI Chat Completions format

OpenAI's Chat Completions streaming sends data: lines with no event: field. Each chunk is a JSON object with object: "chat.completion.chunk" and a choices array containing a delta object. The delta holds either role, content, tool_calls, or refusal — never the full message, only the incremental piece. Source: OpenAI streaming guide, retrieved 2026-07-31.

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Key details:

• Termination: the literal string data: [DONE] signals the stream is complete. This is not valid JSON.
• Usage: only included if you pass stream_options: {"include_usage": true}. The usage chunk has an empty choices array.
• Rate-limit headers: sent in the initial HTTP response headers (x-ratelimit-limit-requests, x-ratelimit-remaining-tokens, etc.), not in the stream. Source: Simon Willison's streaming LLM APIs investigation, retrieved 2026-07-31.

OpenAI's newer Responses API uses typed semantic events (response.created, response.output_text.delta, response.completed) instead of the Chat Completions chunk format, but Chat Completions remains the format that OpenAI-compatible providers emulate.

Anthropic Messages format

Anthropic uses both event: and data: fields in SSE — a key difference from OpenAI. The stream follows a lifecycle: message_start → content_block_start → content_block_delta (repeated) → content_block_stop → message_delta → message_stop. Source: Anthropic streaming documentation, retrieved 2026-07-31.

event: message_start
data: {"type":"message_start","message":{"id":"msg_01X","role":"assistant","content":[],"usage":{"input_tokens":25,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: ping
data: {"type":"ping"}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":15}}

event: message_stop
data: {"type":"message_stop"}

Key details:

• No [DONE]: Anthropic terminates with event: message_stop. There is no data: [DONE] sentinel.
• Ping events: Anthropic sends event: ping keepalives. A client that only expects data: lines will choke.
• Usage in two places: input tokens arrive in message_start; output tokens arrive in message_delta.
• Content block indexing: the index field supports multiple content blocks (text, tool_use, thinking) in a single response. OpenAI uses choices[0].index for a similar but structurally different purpose.
• Rate-limit headers: use anthropic-ratelimit-* prefix, not x-ratelimit-*.

DashScope (OpenAI-compatible mode)

DashScope exposes an OpenAI-compatible endpoint at https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions. When you set stream: true, the response format mirrors OpenAI's Chat Completions chunking: data: lines with chat.completion.chunk objects, terminated by data: [DONE]. Source: Alibaba Cloud Model Studio OpenAI compatibility, retrieved 2026-07-31.

The differences are subtle:

• Authentication header: DashScope's native API uses Authorization: Bearer <api-key> in the OpenAI-compatible mode, identical to OpenAI. Its legacy DashScope-native API used a different header (Authorization: Bearer with a DashScope-format key), but the OpenAI-compatible path accepts the same SDK.
• Thinking/reasoning tokens: When calling Qwen models with thinking enabled (e.g., qwen3.7-max with enable_thinking: true), reasoning tokens stream separately. DashScope wraps them in the same delta structure but with a reasoning_content field alongside content. This field does not exist in OpenAI's spec.
• Chunk size: In our testing, DashScope sends slightly larger token batches per SSE chunk than OpenAI — typically 2–4 tokens per chunk versus OpenAI's 1 token per chunk for text models. This means fewer HTTP frames for the same output length, which can affect time-to-first-token perception in UIs.

DeepSeek format

DeepSeek's API is explicitly OpenAI-compatible and also supports Anthropic-format endpoints at https://api.deepseek.com/anthropic. For the OpenAI-compatible path, streaming follows the same data: + chat.completion.chunk + data: [DONE] pattern. Source: DeepSeek API documentation, retrieved 2026-07-31.

Quirks we observed:

• Reasoning content: Like DashScope, DeepSeek streams reasoning tokens via a reasoning_content field in the delta when thinking mode is enabled. This is the same extension DashScope uses, not in the OpenAI spec.
• Dual format support: DeepSeek is the only first-party provider that natively speaks both OpenAI and Anthropic SSE formats from the same API. The /anthropic path returns Anthropic-style event: + data: lifecycle events.
• Cache hit indicators: DeepSeek may include cache_creation_input_tokens and cache_read_input_tokens in the usage chunk, similar to Anthropic's prompt caching fields.

Token buffering differences

Providers do not flush tokens at the same rate. This matters for user experience — a chat UI rendering character by character looks responsive, but a UI receiving 3-second silence followed by a 50-token dump feels broken.

For reasoning models across all providers, expect a long pause before the first visible content token — the model is generating its chain-of-thought first. Some providers stream reasoning tokens (DashScope, DeepSeek with reasoning_content); others withhold them until the final answer begins.

Mid-stream error handling

When something goes wrong after the SSE connection is already open and tokens have started flowing, each provider handles it differently. This is the hardest part of multi-provider streaming — your HTTP status code is already 200.

OpenAI

OpenAI sends a final chunk with an error field or closes the connection abruptly. There is no standard error event type in their Chat Completions SSE stream. If the model hits a content filter mid-generation, the last delta may contain a finish_reason: "content_filter" instead of "stop". If the server crashes, the TCP connection drops with no data: [DONE] — your client must handle incomplete streams.

Anthropic

Anthropic can send an event: error with a data: {"type":"error","error":{"type":"overloaded_error","message":"..."}} payload. The error arrives as a proper SSE event, so clients listening for event: types can catch it cleanly. The connection then closes. Source: Anthropic API errors documentation, retrieved 2026-07-31.

DashScope

DashScope's OpenAI-compatible streaming follows OpenAI's pattern: errors mid-stream are uncommon but manifest as connection drops or malformed chunks. The native DashScope API has more structured error events, but the compatible-mode path trades that for OpenAI wire compatibility.

DeepSeek

DeepSeek follows OpenAI's error behavior for its OpenAI-compatible path. The Anthropic-compatible path follows Anthropic's error event pattern.

Production rule: Always implement a timeout and an incomplete-stream detector. If you receive chunks but never see the termination signal (data: [DONE] or event: message_stop) within your timeout, treat the response as failed and log the partial output for debugging.

Tool-call streaming: the hardest normalization problem

Streaming tool calls means receiving the function name and JSON arguments character by character. This is already tricky with a single provider. Across providers, the differences multiply.

OpenAI

Tool calls stream via delta.tool_calls[i].function.name (sent once) and delta.tool_calls[i].function.arguments (sent as incremental string fragments). You must concatenate arguments fragments and parse the complete JSON only after finish_reason: "tool_calls". Multiple tool calls can interleave by index.

{"delta":{"tool_calls":[{"index":0,"id":"call_abc","type":"function","function":{"name":"get_weather","arguments":""}}]}}
{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"lo"}}]}}
{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"cation\":"}}]}}
{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"Paris\"}"}}]}}

Anthropic

Anthropic streams tool use as a content block with type: "tool_use". The content_block_start event carries {"type":"tool_use","id":"toolu_abc","name":"get_weather","input":{}}, and then content_block_delta events carry {"type":"input_json_delta","partial_json":"..."} fragments. The JSON fragments follow the same concatenation pattern, but the envelope is completely different from OpenAI.

DashScope and DeepSeek

Both follow OpenAI's tool-call streaming format when using the OpenAI-compatible endpoint. DeepSeek's Anthropic-compatible endpoint follows Anthropic's tool-use streaming format. DashScope does not expose an Anthropic-compatible endpoint.

Gateway challenge: A streaming gateway that accepts any upstream format and outputs a consistent downstream format must maintain per-connection state: which content blocks are open, which tool-call arguments are being assembled, and what the original provider's termination semantics are. The LLM-Rosetta project (Argonne National Laboratory, April 2026) formalized this as a 10-type stream event schema with stateful context management. Source: LLM-Rosetta paper, retrieved 2026-07-31.

Content-Type and connection lifecycle

All providers use HTTP/1.1 chunked transfer encoding or HTTP/2 data frames. The SSE connection is a long-lived HTTP response — it is not a WebSocket. OpenAI's Responses API also offers a WebSocket mode for persistent connections, but that is a separate transport.

Browser EventSource API cannot consume any of these because EventSource only supports GET requests; LLM APIs require POST. Use fetch() with a streaming body reader or a library like @microsoft/fetch-event-source.

Gateway streaming: normalizing diverse formats

When TheRouter routes a streaming request through configured providers, the gateway must normalize upstream SSE into a consistent downstream contract. We route OpenAI-compatible requests through configured providers and support provider/model routing and fallback where the live product path supports it.

The normalization challenge has three layers:

1. Event envelope: Convert Anthropic's event: + data: lifecycle into OpenAI-style data:-only chunks, or vice versa. This includes mapping message_start → first chunk with role, content_block_delta → delta.content, and message_delta → final chunk with finish_reason.
2. Extension fields: Strip or preserve provider-specific fields like reasoning_content, cache_read_input_tokens, or Anthropic's usage placement. Downstream clients should not break on unexpected fields, but they also should not rely on fields that only one upstream provider sends.
3. Termination semantics: Ensure the downstream always receives the expected termination signal regardless of what the upstream sends. If the upstream is Anthropic and the downstream expects OpenAI format, the gateway must emit data: [DONE] after processing event: message_stop.

For a broader view of how providers compare on pricing, models, and API surface, see our LLM API providers comparison and OpenAI-compatible API providers guide.

Code example: minimal cross-provider streaming client

This Python client handles both OpenAI-style and Anthropic-style SSE streams. It is deliberately minimal — production implementations need retry logic, timeout handling, and proper backpressure.

import httpx
import json

def stream_openai_compatible(base_url: str, api_key: str, model: str, messages: list):
    """Stream from any OpenAI-compatible endpoint (OpenAI, DashScope, DeepSeek)."""
    with httpx.stream(
        "POST",
        f"{base_url}/chat/completions",
        headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
        json={"model": model, "messages": messages, "stream": True,
              "stream_options": {"include_usage": True}},
        timeout=60.0,
    ) as response:
        buffer = ""
        for line in response.iter_lines():
            if not line or line.startswith(":"):
                continue
            if line == "data: [DONE]":
                break
            if line.startswith("data: "):
                chunk = json.loads(line[6:])
                delta = chunk.get("choices", [{}])[0].get("delta", {})
                if content := delta.get("content"):
                    yield content
                # Handle reasoning_content for DashScope/DeepSeek thinking models
                if reasoning := delta.get("reasoning_content"):
                    yield f"[thinking] {reasoning}"

def stream_anthropic(api_key: str, model: str, messages: list):
    """Stream from Anthropic's native API."""
    with httpx.stream(
        "POST",
        "https://api.anthropic.com/v1/messages",
        headers={
            "x-api-key": api_key,
            "anthropic-version": "2023-06-01",
            "Content-Type": "application/json",
        },
        json={"model": model, "messages": messages, "stream": True, "max_tokens": 4096},
        timeout=60.0,
    ) as response:
        for line in response.iter_lines():
            if not line:
                continue
            if line.startswith("event: "):
                event_type = line[7:]
                if event_type == "message_stop":
                    break
                continue
            if line.startswith("data: "):
                data = json.loads(line[6:])
                if data.get("type") == "content_block_delta":
                    delta = data.get("delta", {})
                    if delta.get("type") == "text_delta":
                        yield delta.get("text", "")

For a deeper look at how fallback routing works when a streaming provider fails mid-request, see our LLM API fallback routing guide.

Production gotchas

Proxy and CDN buffering

Reverse proxies (nginx, Cloudflare, AWS ALB) can buffer SSE responses and deliver them in large batches instead of token by token. This destroys the streaming experience. Mitigations:

• nginx: proxy_buffering off; and X-Accel-Buffering: no response header
• Cloudflare: use the cf-no-transform header or disable response buffering in the zone
• AWS ALB: ALB does not support SSE natively in all configurations — consider NLB or direct connections

Timeout configuration

SSE connections are long-lived. Default HTTP client timeouts (30s) will kill streaming requests for long outputs. Set separate read timeouts (per-chunk) vs. connection timeouts:

# httpx: total timeout vs. read timeout
timeout = httpx.Timeout(connect=10.0, read=120.0, write=10.0, pool=10.0)

Backpressure

If your client processes tokens slower than the server sends them, the TCP receive buffer fills up and the server's send will eventually block. For most LLM APIs this is not a practical concern — token generation is slower than network transmission — but batch streaming or cached responses can overwhelm slow clients.

Keep-alive and reconnection

SSE connections can drop silently due to network issues. Anthropic's ping events help detect dead connections quickly. For OpenAI-style streams without pings, implement a read timeout: if no data arrives within N seconds and the stream has not terminated, reconnect. Note that LLM APIs generally do not support resuming a stream from where it left off — a dropped connection means a new request.

Checklist: adding a new provider's streaming endpoint

When integrating a new LLM provider's streaming API, verify each of these:

1. Event format: Does the provider use data:-only (OpenAI style) or event: + data: (Anthropic style)?
2. Termination signal: Is it data: [DONE], event: message_stop, or something else?
3. Content extraction path: Where is the text delta? choices[0].delta.content? delta.text_delta.text? Something provider-specific?
4. Tool-call streaming: Does it follow OpenAI's tool_calls[i].function.arguments pattern or Anthropic's input_json_delta pattern?
5. Usage reporting: Is usage included in the stream? Only if requested? In the first chunk, last chunk, or both?
6. Extension fields: Does the provider add non-standard fields like reasoning_content, cache_read_input_tokens, or custom metadata?
7. Error signaling: How does the provider signal errors mid-stream? Structured event? Connection drop? Malformed chunk?
8. Keepalive mechanism: Does the provider send ping events, or is the connection silent between token generations?
9. Chunk size: Does the provider send individual tokens or batch them? This affects perceived latency.

For how errors and status codes differ across providers in non-streaming contexts, see our rate-limit comparison and provider comparison.

FAQ

Can I use the browser EventSource API with LLM streaming endpoints?

No. The EventSource API only supports GET requests. All LLM streaming APIs require POST. Use fetch() with a ReadableStream reader or a library like @microsoft/fetch-event-source. For provider integration details, see our OpenAI-compatible API providers guide.

Do all OpenAI-compatible providers stream identically to OpenAI?

Mostly yes for the basic text streaming path. The data: line format and data: [DONE] termination are consistent. Differences appear in extension fields (reasoning_content on DashScope and DeepSeek), usage reporting behavior, and chunk sizes. Tool-call streaming is the area with the most subtle differences.

What happens if I switch from OpenAI to Anthropic mid-project?

Your streaming client will break. Anthropic uses a completely different event lifecycle (message_start / content_block_delta / message_stop) with named event: types. You need either a new client implementation or a gateway that normalizes Anthropic's format to OpenAI-compatible chunks.

How do I handle reasoning tokens in streaming?

DashScope and DeepSeek stream reasoning tokens via a reasoning_content field in the delta object. Anthropic streams extended thinking as a separate content block with type: "thinking". OpenAI does not stream reasoning tokens in Chat Completions — they are consumed internally. Your client must handle the reasoning_content field gracefully (ignore it if unexpected, display it if your UI supports it).

Is there a standard for cross-provider streaming normalization?

The LLM-Rosetta project (April 2026) proposed a hub-and-spoke IR with 10 stream event types as a formal approach. In practice, most gateways and SDKs (LiteLLM, Portkey, TheRouter) implement their own normalization layers. The OpenAI Chat Completions chunk format is the de facto standard that most compatible providers target.