Cost-per-Outcome AI Routing: How to Move From Token Pricing to Useful Intelligence per Dollar

A model that costs $0.50 per million input tokens sounds cheaper than one at $3.00 — until the cheap model needs three attempts to get the answer right and the expensive one nails it on the first pass. When you factor in retries, human review, and wasted compute, the "expensive" model often costs less per successful task.

This is the core insight behind cost-per-outcome (CPO) routing: optimize for the total cost of producing a successful result, not the unit price of tokens. OpenAI formalized this thinking in July 2026 with their Useful Intelligence per Dollar scorecard. We built this reference to translate that framework into practical routing decisions.

Why Cost-per-Token Is the Wrong Routing Metric

OpenAI-compatible means a provider exposes a chat-completions endpoint whose request and response shape matches the OpenAI API contract closely enough that an unmodified OpenAI SDK call works against it after swapping three values: API key, base URL, and model name. The minimum surface in practice is POST /v1/chat/completions with messages, model, and an OpenAI-shaped streaming response.

Token pricing tells you the cost of raw material. It does not tell you the cost of the finished product.

Consider a concrete example: you route a code review task to two different models.

Model B costs 6× more per token but delivers 4× lower cost per successful outcome. The token price comparison was never the right question.

This pattern repeats across every workload we've measured. The "verbosity tax" — cheaper models generating 15–20% more tokens to convey the same information — erodes unit-price advantages. Retry loops compound the problem. And human review time, which rarely appears in token-cost spreadsheets, often dominates total cost.

The Four Metrics: OpenAI's Useful Intelligence per Dollar Scorecard

OpenAI CFO Sarah Friar proposed a scorecard that asks four questions. We've adapted each metric for gateway operators who need to make routing decisions.

1. Useful Work Completed

What it measures: How many tasks did AI actually finish at the required quality bar?

Tokens create value only when they transform into work people can use. For a support team, "done" means a customer issue resolved. For an engineering team, it means a code change that passes tests. For a legal team, it means a contract reviewed accurately.

How to track it: Define "done" per workflow. Measure completions in the system where the work happens — your ticketing system, CI pipeline, or document management tool — not in your AI dashboard.

Routing implication: A model with a higher task completion rate on your specific workload may justify a higher token price. Completion rate varies by task type, so measure per-workflow, not as a global average.

2. Cost per Successful Task

What it measures: The full cost of completing one unit of work at the required quality level.

The formula:

CPO = (Token Cost + Retrieval Cost + Orchestration Cost + Human Review Cost) / Accepted Outcomes

Where:

• Token Cost = (input tokens × input price + output tokens × output price) × average attempts
• Retrieval Cost = embedding calls + vector DB queries per task
• Orchestration Cost = gateway compute, logging, queue overhead
• Human Review Cost = review time × hourly rate × escalation rate

Routing implication: This is the number that should drive model selection. Compare CPO across providers, not token prices. A frontier model like GPT-5.6 Sol or Claude Opus 4 may deliver the best CPO for complex tasks despite premium pricing, because it needs fewer attempts and less review.

3. Dependability Rate

What it measures: How often does the AI output meet the quality bar without human intervention?

Track three outcomes per task:

• Ready to use: Output met quality bar as delivered
• Needs correction: Required another attempt or human edits
• Needs escalation: A person had to step in and finish the work

Routing implication: Dependability directly affects CPO. A model with 90% ready-to-use rate has 10% waste; a model with 70% has 30% waste. When you route complex tasks to more dependable (often more expensive) models, the reduced waste typically outweighs the higher token cost.

4. Value at Scale

What it measures: Does each AI dollar produce more value as usage grows?

Track the same workflow over time:

• Does CPO decrease as the team gains experience with prompts and routing?
• Does the provider improve model capability without proportional price increases?
• Do caching and routing optimizations compound?

Routing implication: Providers with prompt caching (OpenAI, Anthropic, DashScope) create scale advantages: repeated context costs less per call. Routing through a gateway with caching awareness amplifies this effect.

Token-Based vs. Outcome-Based Routing: A Comprehensive Comparison

Decision Tree: When to Optimize for What

Use this decision tree to choose your routing strategy for each workflow:

Step 1 — Is the task output directly used without human review?

• Yes → Optimize for dependability (pick the model with highest ready-to-use rate)
• No → Go to Step 2

Step 2 — Is the task high-volume (>1,000 calls/day)?

• Yes → Go to Step 3
• No → Optimize for CPO (pick the model with lowest total cost per accepted outcome)

Step 3 — Is the task uniform (same format, predictable complexity)?

• Yes → Optimize for throughput + token cost (a cheaper, faster model is likely sufficient)
• No → Optimize for CPO with tiered routing (route simple variants to cheap models, complex variants to capable models)

Step 4 — Does the workflow benefit from prompt caching?

• Yes → Weight providers with caching support (OpenAI, Anthropic, DashScope) and factor cache-hit savings into CPO
• No → Compare CPO at list prices

Practical Implementation: Configuring Routing for Outcome-Based Selection

Moving from token-based to outcome-based routing does not require new infrastructure. It requires new measurement and different routing logic.

Step 1: Instrument your baseline

Before changing routing rules, measure current CPO for your top 3 workflows:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.therouter.ai/v1",  # or your gateway
    api_key="your-key"
)

# Track per-request metadata for CPO calculation
import time

def measure_task(prompt, expected_schema):
    attempts = 0
    total_tokens = 0
    start = time.time()

    while attempts < 5:
        attempts += 1
        response = client.chat.completions.create(
            model="openai/gpt-5.5",
            messages=[{"role": "user", "content": prompt}],
        )
        total_tokens += response.usage.total_tokens
        result = response.choices[0].message.content

        if validate(result, expected_schema):
            return {
                "success": True,
                "attempts": attempts,
                "total_tokens": total_tokens,
                "latency_ms": (time.time() - start) * 1000,
            }

    return {"success": False, "attempts": attempts, "total_tokens": total_tokens}

Step 2: Compare CPO across models

Run the same task set against multiple models and calculate CPO for each:

models_to_test = [
    "openai/gpt-5.5",        # Frontier
    "openai/gpt-5.4-mini",   # Mid-tier
    "deepseek/deepseek-v4-flash", # Cost-efficient
    "qwen/qwen3.7-max",      # Chinese provider option
]

for model in models_to_test:
    results = [measure_task(prompt, schema) for prompt in test_prompts]
    successes = [r for r in results if r["success"]]
    total_cost = sum(r["total_tokens"] * get_price(model) for r in results)
    cpo = total_cost / len(successes) if successes else float("inf")
    print(f"{model}: CPO=${cpo:.4f}, success_rate={len(successes)/len(results):.1%}")

Step 3: Configure tiered routing based on CPO data

Once you have CPO data, configure your routing to send tasks to the model with the lowest CPO for each task type — not the cheapest per token.

For complex reasoning tasks, a frontier model (Claude Opus, GPT-5.6 Sol) typically has the lowest CPO despite higher token prices. For simple classification or extraction, a cost-efficient model (DeepSeek V4, Qwen3-7-Plus) wins on CPO because the task doesn't benefit from stronger reasoning.

Step 4: Add fallback routing for dependability

Fallback routing directly improves CPO by reducing failed attempts. When your primary model is rate-limited or unavailable, the request falls back to a secondary model rather than failing and requiring a retry.

Configure fallback chains based on CPO rank order, not price order. If Model A has the lowest CPO but goes down, fall back to Model B (second-lowest CPO), not Model C (cheapest per token).

The Verbosity Tax and Why It Matters for Routing

One frequently overlooked cost factor: cheaper models often generate more tokens to express the same information. Research from production deployments shows that budget models produce 15–20% more output tokens on average for equivalent tasks.

This "verbosity tax" erodes token-price advantages:

The capable model costs 5.3× more per task by output tokens alone — but if it completes in 1 attempt vs. 3, the real cost is 5.3× / 3 = 1.8× with a third of the latency. Add human review savings and the capable model often wins.

Prompt Caching and Scale Advantages

Providers with prompt caching offer a structural CPO advantage at scale. When repeated context (system prompts, few-shot examples, retrieved documents) is cached, subsequent calls pay a fraction of the input token cost.

For high-volume workflows with shared context, caching can cut input costs by 50–90%. This directly reduces the numerator in your CPO formula. See our prompt caching guide for implementation details.

FAQ

What is cost-per-outcome in AI routing?

Cost-per-outcome (CPO) measures the total cost of producing one successful AI task completion — including tokens, retries, human review, and orchestration overhead — rather than just the price per token. It is the metric that directly maps to business value: you care about what a successful result costs, not what a token costs.

Why is cost-per-token misleading for AI cost optimization?

A cheaper model per token may require more attempts, produce more verbose output, or need more human correction to reach the same result. The total cost of a successful outcome can be higher with a "cheap" model than with a more capable one that completes the task in one pass. OpenAI's Useful Intelligence per Dollar scorecard makes this case explicitly: "the lowest price per token does not always produce the lowest cost per outcome."

How does routing configuration affect cost-per-outcome?

Routing determines which model handles each request. By routing simple tasks to cheaper models and complex tasks to more capable ones, you minimize CPO across your entire workload. Fallback routing adds dependability: when your primary model is unavailable, requests automatically fall to a secondary, reducing failed attempts. The net effect is lower CPO than any single-model strategy.

What is OpenAI's Useful Intelligence per Dollar framework?

OpenAI CFO Sarah Friar proposed this scorecard on July 17, 2026. It measures four dimensions: (1) useful work completed, (2) cost per successful task, (3) dependability rate, and (4) value at scale. The framework reframes AI ROI from infrastructure cost to business outcome value, arguing that token pricing alone cannot capture whether AI spending is producing real returns.

How do I calculate cost-per-outcome for my AI workloads?

CPO = Total Workflow Cost / Number of Accepted Outcomes

Total Workflow Cost = model tokens (input + output, including retries) + retrieval/embedding costs + orchestration overhead + human review time × hourly rate. Count only tasks that met your quality bar as accepted outcomes. A task that required 3 attempts counts the token cost of all 3 attempts in the numerator but only 1 in the denominator.

Can model fallback routing reduce cost-per-outcome?

Yes. Fallback routing sends requests to a secondary model when the primary is unavailable or rate-limited. This reduces failed attempts and retries, which directly lowers the denominator penalty in CPO calculations. Configure fallback chains by CPO rank, not by token price. See our fallback routing guide for configuration details.

How does TheRouter support outcome-based routing?

TheRouter routes OpenAI-compatible requests through configured providers and supports provider/model routing and fallback when live product paths support it. You can configure primary and fallback models per route, enabling tiered routing based on your CPO measurements. Combined with unified billing surfaces, this lets you track cost by workflow rather than by raw token spend.

Sources

All claims in this post are sourced:

• OpenAI, "A scorecard for the AI age" — Retrieved 2026-07-21
• OpenAI API Pricing — developers.openai.com — Retrieved 2026-07-21
• Anthropic Claude Pricing — platform.claude.com — Retrieved 2026-07-21
• PromptMetrics, "AI Pricing in 2026: Why Cost-Per-Outcome Beats Tokens" — Retrieved 2026-07-21
• Axios, "OpenAI's CFO pitches a new way to measure AI's value" — Retrieved 2026-07-21
• CIO Dive, "OpenAI pushes new yardstick for measuring AI investments" — Retrieved 2026-07-21