Multimodal Input Across LLM API Providers: Image, Video, Audio, and File Formats Each Provider Actually Accepts

The fastest answer: OpenAI and Anthropic accept images (JPEG, PNG, GIF, WebP) via URL or base64 with model-specific size limits; DashScope goes further with native video and audio input on Qwen-VL/Omni models; DeepSeek's hosted API does not support image input natively; SiliconFlow hosts multimodal open-weight models with OpenAI-compatible image input. The real complexity is in the details — maximum dimensions, token costs, the detail parameter, and which modalities each model actually handles.

OpenAI-compatible means a provider exposes a chat-completions endpoint whose request and response shape matches the OpenAI API contract closely enough that an unmodified OpenAI SDK call works against it after swapping three values: API key, base URL, and model name. The minimum surface in practice is POST /v1/chat/completions with messages, model, and an OpenAI-shaped streaming response.

Sources: OpenAI Images and Vision, retrieved 2026-08-06; Anthropic Claude Vision, retrieved 2026-08-06; DashScope OpenAI Compatibility, retrieved 2026-08-06; DeepSeek API Docs, retrieved 2026-08-06; SiliconFlow Models, retrieved 2026-08-06.

TL;DR — Multimodal Input Support Table

OpenAI: The Broadest Multimodal Surface

OpenAI offers the most mature multimodal input surface. Vision-capable models (GPT-5.6, GPT-5.5 Pro, GPT-5.4 Mini) accept images through three delivery methods.

Image input methods

URL reference — pass a publicly accessible URL in the image_url field:

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-5.6",
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "What's in this image?"},
            {
                "type": "input_image",
                "image_url": "https://example.com/photo.jpg",
                "detail": "auto"
            }
        ]
    }]
)

Base64 data URI — embed the image directly:

import base64

with open("photo.jpg", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = client.responses.create(
    model="gpt-5.6",
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "Describe this image."},
            {
                "type": "input_image",
                "image_url": f"data:image/jpeg;base64,{b64}",
                "detail": "high"
            }
        ]
    }]
)

File API file_id — upload once, reference many times:

file = client.files.create(file=open("photo.jpg", "rb"), purpose="vision")
# Then reference file.id in the content array

The detail parameter

The detail parameter controls how the model processes images and directly affects token cost:

Supported formats and limits

• Formats: JPEG, PNG, GIF (first frame only), WebP
• Max file size: 20 MB
• Max image dimension: 2048px (long edge, auto-scaled)
• Multiple images: Supported in a single request
• Audio: Dedicated audio models (GPT Transcribe) for speech-to-text
• Video: Not native — extract frames and pass as multiple images

Anthropic Claude: Images and Native PDF

Anthropic Claude (Opus 4.8, Sonnet 5, Haiku) supports image input and is one of the few providers with native PDF processing.

Image input

Claude accepts images as image content blocks with source specifying either base64 or URL:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5-20260714",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/jpeg",
                    "data": "<base64-encoded-data>"
                }
            },
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }]
)

Format limits

• Formats: JPEG, PNG, GIF (non-animated), WebP
• Max file size: 5 MB per image (base64-encoded payload is larger than the raw file)
• Max dimensions: 2576px / 3.75 megapixels (high-res mode, on by default)
• Multiple images: Up to 20 images per request
• Token counting: Based on image dimensions — roughly (width × height) / 750 tokens

Native PDF support

Claude is unique among major providers in offering native PDF input:

• Max pages: 100 per document
• Max file size: 32 MB
• Delivery: Base64-encoded in the content array with media_type: "application/pdf"
• Token cost: Each page is treated as approximately one image for token counting

What Claude does NOT support

• Video input: Not supported
• Audio input: Not supported
• File upload API: No persistent file storage — base64 or URL per request

DashScope (Qwen): The Widest Modality Coverage

DashScope offers the broadest set of input modalities through the Qwen model family. Via the OpenAI-compatible endpoint, Qwen-VL and Qwen-Omni models accept images, video, and audio.

Image input via OpenAI-compatible API

from openai import OpenAI

client = OpenAI(
    api_key="sk-xxx",
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1"
)

response = client.chat.completions.create(
    model="qwen-vl-max",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
        ]
    }]
)

Supported models and modalities

Video input

DashScope Qwen-VL models accept video URLs directly in the content array. The model extracts frames internally:

response = client.chat.completions.create(
    model="qwen-vl-max",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe what happens in this video."},
            {"type": "video_url", "video_url": {"url": "https://example.com/clip.mp4"}}
        ]
    }]
)

Audio input

Qwen3.5-Omni accepts audio input for speech understanding and can produce audio output:

• Formats: WAV, MP3, FLAC, OGG
• Max duration: Model-dependent
• Delivery: URL reference

Image format details

• Formats: JPEG, PNG, BMP, WebP, TIFF
• Max dimensions: Model-dependent (up to megapixel-level for VL-Max)
• Delivery: URL or base64
• Multiple images: Supported

DeepSeek: No Native Vision on Hosted API

As of August 2026, DeepSeek's hosted V4-Flash and V4-Pro models do not support native image input through the official API at api.deepseek.com. The DeepSeek API documentation describes V4 as text-only for the hosted service.

What is available

• DeepSeek-VL (open-weight): An older vision-language model available for self-hosting, but not served through the official API
• Third-party hosted: Platforms like Fireworks AI and SiliconFlow host multimodal DeepSeek variants with vision support via document inlining
• No video/audio/PDF: None of these modalities are supported on the hosted API

Routing implication

If your application needs both DeepSeek's cost efficiency for text tasks and vision capabilities, you need a routing layer that can direct multimodal requests to a vision-capable provider while keeping text-only requests on DeepSeek. This is exactly the kind of heterogeneous provider topology that a routing layer handles.

SiliconFlow: Open-Weight Multimodal Models

SiliconFlow hosts 200+ models including multimodal open-weight models with OpenAI-compatible API endpoints.

Image input

SiliconFlow supports image input through hosted vision models (Qwen-VL, InternVL, GLM-4V) using the standard OpenAI-compatible format:

from openai import OpenAI

client = OpenAI(
    api_key="sk-xxx",
    base_url="https://api.siliconflow.cn/v1"
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-72B-Instruct",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
        ]
    }]
)

Format details

• Formats: JPEG, PNG, WebP
• Max dimensions: Megapixel-level (model-dependent)
• Delivery: URL or base64
• No video/audio/PDF: These modalities are not supported through the API

Common Gotchas: Format Conversion and Size Limits

Base64 overhead

Base64 encoding increases file size by approximately 33%. A 4 MB JPEG becomes approximately 5.3 MB when base64-encoded. This matters most for Anthropic's 5 MB limit — your source image should be under approximately 3.75 MB to fit within the encoded limit.

URL accessibility

When passing image URLs, the URL must be publicly accessible. Private endpoints, VNet-restricted URLs, and URLs behind authentication will fail silently or return errors. Consider using base64 for images from private sources.

Dimension auto-scaling

Both OpenAI and Anthropic auto-scale images that exceed their maximum dimensions. OpenAI scales to fit within 2048px on the long edge. Anthropic scales to fit within 2576px or 3.75 megapixels. Scaling happens server-side, so you pay tokens on the scaled dimensions, not the original.

GIF handling

OpenAI processes only the first frame of animated GIFs. Anthropic does not support animated GIFs at all. If you need to analyze animations, extract key frames and pass them as multiple images.

Token cost implications

Image tokens are billed at the same rate as text tokens. A single high-detail image on OpenAI can cost 1,000+ tokens. On Anthropic, a 1920×1080 image costs approximately 2,764 tokens. Plan your multimodal request budgets accordingly.

Decision Tree: Which Provider for Which Modality

Need image understanding?
├── Budget priority → SiliconFlow (free-tier VL models)
├── Accuracy priority → OpenAI GPT-5.6 or Anthropic Opus 4.8
├── China deployment → DashScope Qwen-VL
└── Cost-optimized text + vision routing → TheRouter (route vision to capable provider, text to cheapest)

Need video understanding?
├── DashScope Qwen-VL-Max or Qwen3.5-Omni
└── OpenAI (extract frames manually)

Need audio input?
├── DashScope Qwen3.5-Omni (speech understanding)
└── OpenAI GPT Transcribe (speech-to-text)

Need PDF processing?
├── Anthropic Claude (native, up to 100 pages)
├── OpenAI (via File API)
└── DashScope (via extraction endpoint)

TheRouter Note: Routing Multimodal Requests Across Providers

When you route OpenAI-compatible requests through TheRouter, multimodal requests follow the same routing rules as text requests. The key consideration is that not all providers support all modalities — DeepSeek's hosted API, for example, does not accept image input.

We recommend configuring model fallbacks so that multimodal requests route to vision-capable providers while text-only requests can take advantage of the most cost-effective option. This is especially useful when your application handles a mix of text-only and multimodal workloads.

FAQ

Q: Can I send multiple images in one API request? A: Yes — OpenAI, Anthropic, DashScope, and SiliconFlow all support multiple images per request. OpenAI and Anthropic include them in the content array. Keep in mind that each image counts toward your token budget. See OpenAI's vision guide and the respective Anthropic and DashScope provider pages.

Q: What happens if my image exceeds the size limit? A: OpenAI returns a 400 error for images over 20 MB. Anthropic returns an error for base64 payloads over 5 MB. DashScope and SiliconFlow have model-specific limits. Resize images client-side before sending. See SiliconFlow for model-specific limits.

Q: Is there a way to reduce image token costs? A: On OpenAI, set detail: "low" to use a fixed 85-token budget per image. On Anthropic, resize images to smaller dimensions before encoding. Across all providers, JPEG compression reduces file size without significantly impacting model understanding. For more on cost optimization, see our cost optimization routing guide.

Q: Can I use the OpenAI SDK to send images to DashScope? A: Yes. DashScope's OpenAI-compatible endpoint accepts the same image_url content block format. Change the base_url and API key, and use a Qwen-VL model name. See our DashScope API guide for setup details.

Q: Which provider has the best image understanding accuracy? A: For general image understanding, OpenAI GPT-5.6 and Anthropic Claude Opus 4.8 consistently rank highest in benchmarks. For cost-efficient image understanding, DashScope Qwen3.8-Max and SiliconFlow-hosted Qwen-VL models offer strong performance at lower price points. Check the models page for current benchmark comparisons.