LLM API Observability Tools Compared 2026: Langfuse, Helicone, Portkey, Arize Phoenix, and What to Track Through Your Gateway

Pick Langfuse when you want open-source, self-hostable, framework-agnostic tracing with OpenTelemetry support. Pick Helicone when you want the fastest possible setup — one URL change — with built-in cost dashboards and caching. Pick Portkey when gateway routing and observability need to live in the same control plane. Pick Arize Phoenix when evaluation is the priority and you want notebook-first workflows with zero feature gates. Pick LangSmith when your stack is LangChain-native and you want annotation queues that close the human-review loop. We compared six tools across the dimensions that actually matter in production: tracing depth, evaluation maturity, cost tracking, self-hosting, alerting, and pricing predictability.

OpenAI-compatible means a provider exposes a chat-completions endpoint whose request and response shape matches the OpenAI API contract closely enough that an unmodified OpenAI SDK call works against it after swapping three values: API key, base URL, and model name. The minimum surface in practice is POST /v1/chat/completions with messages, model, and an OpenAI-shaped streaming response.

Sources: Langfuse GitHub, retrieved 2026-07-28; Helicone GitHub, retrieved 2026-07-28; Portkey Pricing, retrieved 2026-07-28; Arize Phoenix GitHub, retrieved 2026-07-28; LangSmith — LangChain, retrieved 2026-07-28; Confident AI — 11 LLM Observability Tools 2026, retrieved 2026-07-28; Firecrawl — Best LLM Observability Tools 2026, retrieved 2026-07-28.

TL;DR — Observability Tools at a Glance

The table answers the first question most teams ask: "Can I self-host it, and what does it cost at 500K requests per month?" After that, the comparison gets more nuanced.

What LLM Observability Actually Means

Traditional APM (Datadog, New Relic) tracks latency, error rates, and infrastructure health. LLM observability adds three layers that APM tools were not designed for:

1. Trace-level visibility — every LLM call, tool invocation, retrieval step, and intermediate reasoning in a multi-step agent chain, not just the HTTP request/response envelope.
2. Cost attribution — token counts, cache hits, per-model pricing, and per-user or per-feature spend breakdowns. LLM API costs compound faster than traditional compute because pricing is per-token, not per-instance.
3. Output quality scoring — faithfulness, relevance, hallucination detection, safety checks. A 200 OK from the provider tells you nothing about whether the response was correct.

If your stack only logs requests and responses, you have monitoring. Observability starts when you can reconstruct why a specific output was produced and whether it was good.

Langfuse — Open-Source All-in-One

Langfuse is the open-source leader with 28,000+ GitHub stars and an MIT license. It covers tracing, prompt versioning, evaluation (LLM-as-judge, user feedback, custom metrics), and a built-in playground — all self-hostable on Postgres + ClickHouse.

What we like:

• Framework-agnostic. Native SDKs for Python and JavaScript, plus connectors for LangChain, LlamaIndex, and 50+ frameworks. OpenTelemetry support means traces can flow into your existing stack.
• Self-hosting is well-documented and actively maintained. No feature gates between cloud and self-hosted.
• Prompt management with version control and A/B comparison.
• Multi-turn conversation tracing with cost attribution per turn.

What to watch:

• The UI is functional but less polished than some commercial alternatives.
• Evaluation features are improving rapidly but are not as deep as purpose-built eval tools (Arize Phoenix, Confident AI).
• Cloud free tier caps at 25K spans/month — sufficient for development, tight for production.

Pricing: Free (25K spans/mo, unlimited team members, 60-day retention). Pro from ~$39/mo for 100K spans. Self-hosted is free and unrestricted.

When to pick Langfuse: You want one platform for tracing + evals + prompt management, you value self-hosting and open source, and your team uses multiple frameworks.

Helicone — Fastest Setup, Gateway-Native

Helicone takes the proxy approach: change your base URL (or add a header), and every LLM request flows through Helicone's gateway. No SDK integration, no code changes beyond the URL. This makes it the fastest observability setup in the category.

What we like:

• One-line integration. Replace api.openai.com with oai.helicone.ai and you are logging requests, costs, latency, and token usage immediately.
• The largest open-source API pricing database (300+ models). Cost tracking is accurate across providers because Helicone maintains the pricing data.
• Built-in caching and rate limiting reduce API spend without application changes.
• Prompt experimentation and versioning through the dashboard.

What to watch:

• Observability depth is request-level, not span-level. If you need step-by-step agent trace trees, Helicone shows you the LLM calls but not the application logic between them.
• Evaluation is limited compared to Langfuse or Arize Phoenix — Helicone scores requests but does not offer deep faithfulness/hallucination metrics.
• The proxy architecture means all traffic routes through Helicone (or your self-hosted instance). Teams with strict data-residency requirements should self-host.

Pricing: Free for unlimited personal use (no corporate). Pro at $79/mo. Team at $799/mo. Self-hosting available via Docker.

When to pick Helicone: You want cost tracking and request logging with zero code changes, caching matters, and you do not need deep agent tracing or evaluation.

Portkey — Gateway + Observability in One

Portkey started as an AI gateway (routing, fallbacks, load balancing, guardrails) and expanded into observability. In 2026, Portkey open-sourced its gateway (MIT, "Gateway 2.0") so routing, governance, observability, and cost controls can all run self-hosted.

What we like:

• Routing and observability live in the same control plane. When a fallback fires, you see the trace of why it fired and which provider served the request.
• Config-driven architecture. Fallback chains, load balancing, and guardrails are defined in config, not code — making them auditable and version-controllable.
• 250+ model integrations with 20–40ms gateway overhead.
• Semantic caching, RBAC, and budget controls built in.

What to watch:

• Observability is provider-level and request-level, not output-quality-level. Portkey tells you which model answered and how much it cost, but does not score whether the answer was faithful or relevant.
• Log retention on the free tier is limited. At 100K logs/month, production workloads will hit the ceiling quickly.
• The gateway architecture is a strength for routing but adds a dependency in the request path.

Pricing: Free (10K logs/mo). Developer at $49/mo (100K logs, 30-day retention, $9 per additional 100K). Enterprise custom.

When to pick Portkey: Your primary need is routing + fallback + observability in one tool, and output-quality evaluation can happen downstream.

Arize Phoenix — Evaluation-First, Notebook-Native

Arize Phoenix is the open-source (ELv2 license) observability and evaluation tool from Arize AI. It was built around the notebook workflow: prompt experimentation, LLM-as-judge evaluation, dataset management, and tracing all happen in a Python-first environment.

What we like:

• Zero feature gates. Every capability (tracing, evaluation, experiments) is available in the open-source version.
• OpenTelemetry-native. Traces use standard OTEL spans, so they integrate with existing observability infrastructure.
• Built-in evaluators for faithfulness, relevance, toxicity, hallucination, and RAG retrieval quality — ready to use without writing custom scoring functions.
• Dataset management and experiment tracking let you version-control evaluation sets alongside prompts.

What to watch:

• The notebook-first philosophy means the UI is secondary. Teams that want a production dashboard with alerting and team collaboration may find the self-hosted UI basic.
• ELv2 license allows self-hosting but has restrictions on offering Phoenix as a managed service. For internal use, this is not a concern.
• Arize's managed cloud starts at $50/mo for teams that do not want to self-host.

Pricing: Phoenix is free to self-host. Arize managed cloud from $50/mo.

When to pick Arize Phoenix: Evaluation depth is your primary requirement, you work in notebooks, and you want OTEL-native tracing with no vendor lock-in.

LangSmith — Best for LangChain Stacks

LangSmith offers the deepest integration with LangChain and LangGraph. Set one environment variable and tracing works automatically — chains, agents, tool calls, and retriever steps are captured without code changes.

What we like:

• Annotation queues route flagged traces (e.g., low-confidence LLM-judge scores) to human reviewers. Reviewers submit structured feedback that flows back into evaluation datasets. This is one of the best human-in-the-loop implementations available.
• Conversation clustering surfaces common user intents from production traffic.
• Agent-specific metrics: tool popularity, error rates, step-level latency.
• LLM-as-judge evaluators for automated scoring of historical runs.

What to watch:

• Outside the LangChain ecosystem, integration depth drops significantly. Framework-agnostic tracing works but requires more manual setup.
• Not open source. Enterprise self-hosting is available on Kubernetes but is not a community-driven project.
• Free tier caps at 5K traces/month — the most restrictive free tier in this comparison.

Pricing: Free (5K traces/mo). Plus at $39/seat/mo (10K traces included). Enterprise self-hosting available.

When to pick LangSmith: Your application is built on LangChain/LangGraph and you want annotation queues and agent debugging with minimal setup.

LangWatch — Regulated and Multi-Agent

LangWatch targets regulated industries and multi-agent architectures. It combines tracing, online evaluation, guardrails, and a production-to-simulation loop — trace production traffic, then replay it through simulated scenarios to test behavior changes before deploying.

What we like:

• Multi-agent span trees with step-level detail.
• Online evaluation runs scoring functions against production traffic in real time.
• Guardrails (content filtering, PII detection) built into the tracing pipeline.
• Self-hostable (Apache-2.0) for data-residency requirements.

What to watch:

• Infrastructure APM is out of scope — LangWatch focuses on LLM application behavior, not the servers running it.
• Smaller community than Langfuse or LangSmith.
• Pricing in euros; starts at ~€29/seat/mo after the free tier.

When to pick LangWatch: You operate in a regulated environment, need guardrails integrated into the observability pipeline, or run multi-agent systems.

Decision Matrix: Pick the Right Tool

Most production stacks end up combining two tools: one for tracing/logging (Langfuse, Helicone, or Portkey) and one for evaluation (Arize Phoenix or a custom pipeline). The tools are not mutually exclusive — OpenTelemetry compatibility means traces from one tool can feed into another.

How API Gateways Fit Into the Observability Stack

If you route LLM requests through a gateway — whether that is Portkey, LiteLLM, or TheRouter — the gateway is a natural instrumentation point.

A routing gateway that sits between your application and upstream providers can emit:

• Request/response logs with provider attribution (which model, which provider, latency, token count).
• Fallback traces showing when and why a request was rerouted to a secondary provider.
• Cost accounting aggregated per provider, per model, and per API key.

TheRouter routes OpenAI-compatible requests through configured providers and supports provider/model routing and fallback when live product paths support it. It provides unified billing/accounting surfaces where implemented. These logs become the foundation for cost observability without requiring a separate proxy in the request path.

The important distinction: a gateway handles operational observability (which provider, what cost, did fallback fire). Output quality observability (was the response faithful, relevant, safe) requires a separate tool like Langfuse, Arize Phoenix, or your own evaluation pipeline. The two complement each other.

For teams already using a gateway, adding Langfuse or Arize Phoenix for quality evaluation creates a complete observability stack:

Application → Gateway (routing + cost logs) → Provider
                ↓
         Tracing tool (Langfuse / Phoenix)
                ↓
         Evaluation pipeline (quality scoring)

This architecture avoids duplicating the proxy layer: the gateway handles routing and cost tracking, while the tracing tool handles span trees and evaluation.

Pricing Comparison at Scale

What does observability cost at 500K LLM requests per month?

Self-hosting Langfuse or Arize Phoenix is the most cost-effective option for high-volume workloads. For teams that prefer managed services, Portkey and Helicone offer predictable per-request pricing.

What to Track: The Observability Checklist

Regardless of which tool you pick, these are the metrics that matter for LLM API observability in production:

1. Latency per model per provider — including time-to-first-token for streaming. Gateway-level routing decisions depend on this.
2. Token usage and cost — input tokens, output tokens, cached tokens, cost per request. Break down by feature, user, or team.
3. Error rates by type — distinguish 429 (rate limit) from 500 (provider error) from 400 (bad request). Each has a different response: retry, fallback, or fix the prompt.
4. Fallback frequency — if your gateway is rerouting 20% of traffic, something is wrong upstream. Track fallback rate as a health signal.
5. Output quality scores — faithfulness, relevance, hallucination rate. Even a sample-based evaluation pipeline (scoring 5–10% of production traffic) catches drift early.
6. Cache hit rate — if you use prompt caching (OpenAI, Anthropic, DashScope) or gateway-level caching (Helicone, Portkey), track the savings.
7. Eval score drift — quality scores over time, segmented by prompt version, model, or use case. A prompt change that improves average quality but degrades one segment is a regression.

FAQ

Q: Do I need a separate observability tool if my gateway already logs requests? Gateway logs cover operational observability: which provider, what cost, did fallback fire. They do not cover output quality. For production applications, you want both.

Q: Can I use multiple observability tools together? Yes. OpenTelemetry compatibility means traces from Langfuse can feed into Arize Phoenix for evaluation, or Helicone's request logs can be enriched with quality scores from a separate pipeline. Most production stacks combine a logging/tracing layer with an evaluation layer.

Q: Which tool is best for a small team just starting out? Langfuse (self-hosted or cloud free tier) gives you the broadest feature set with the lowest commitment. If you want zero-code setup and cost tracking is the priority, start with Helicone.

Q: What about Datadog or New Relic for LLM observability? Both have added LLM observability features. If you are already paying for Datadog or New Relic, their LLM tabs add token and latency tracking alongside your existing infrastructure monitoring. For deeper LLM-specific features (evaluation, prompt management, multi-turn tracing), a purpose-built tool is more capable.

Q: Is self-hosting worth it? At high volume (500K+ requests/month), self-hosting Langfuse or Arize Phoenix saves significant cost and gives you full data control. Below 100K requests/month, managed cloud tiers are simpler and often cheaper than running your own infrastructure.

---

Related: Unified LLM API Providers and AI Gateway Comparison 2026 · LLM API Cost Optimization and Routing Strategies 2026 · AI API Rate Limit Comparison 2026

Internal links: OpenAI provider · Anthropic provider · DashScope provider · SiliconFlow provider · OpenAI-compatible API overview