Reliability and RoutingJuly 30, 2026Flatkey Team

LLM Observability Tools Compared (October 2026): Langfuse, LangSmith, Arize, Helicone, Braintrust, Datadog

Side-by-side comparison of LLM observability and monitoring tools with October 2026 pricing, free tiers, open-source options, a selection guide, and OpenTelemetry setup code.

LLM Observability Tools Compared (October 2026): Langfuse, LangSmith, Arize, Helicone, Braintrust, Datadog

The best LLM observability tools in October 2026 are Langfuse (open source, from $0 to $29/month for managed cloud), LangSmith (best for LangChain and LangGraph teams, $39/seat on Plus), Arize Phoenix and Arize AX (OpenTelemetry-native tracing plus evaluation), Helicone (proxy-based request logging, Pro from $79/month), Braintrust (evaluation-first, Pro $249/month), and Datadog LLM Observability (for teams already on Datadog, billed per LLM span). Pick by operating model first: open-source self-hosting, an AI-native SaaS, a proxy/gateway logger, or your existing APM.

Whatever you pick, LLM observability tools must explain a user-visible AI outcome from the request boundary through retrieval, model attempts, tool calls, validation, and cost. The practical question is: what happened, why did it happen, did the user receive a valid result, and what did that result cost? Start with one trace per user task, a child span for every model attempt, a few outcome measures, and an explicit privacy policy.

Best LLM observability tools at a glance

Pricing below was checked on the vendors' official pricing pages on October 28, 2026. Plans change often, so confirm before you buy.

Tool	Type	Free tier (Oct 2026)	First paid plan (Oct 2026)	Open source / self-host	Best for
Langfuse	AI-native tracing, prompts, evals	Hobby: 50k units/month	Core $29/month with 100k units, then $8 per 100k units; Pro $199/month	Yes (MIT core, self-host)	Teams that want open source with a managed option
LangSmith	Tracing, evals, agent deployment	Developer: $0, 1 seat, 5k base traces/month	Plus $39/seat/month with 10k base traces/month, then pay-as-you-go	Self-host and hybrid on Enterprise	LangChain / LangGraph teams
Arize Phoenix / Arize AX	OpenTelemetry/OpenInference tracing and evaluation	Phoenix OSS free; AX Free: 25k spans/month, 1 GB	AX Pro $50/month with 50k spans/month, 10 GB	Phoenix is open source	Evaluation-heavy teams on OpenTelemetry
Helicone	Proxy / gateway request logging	Hobby: 10,000 requests, 1 GB, 1 seat	Pro $79/month, unlimited seats, usage-based beyond free	Yes (GitHub)	Fast setup by changing the base URL
Braintrust	Evals plus production traces	$0/month, 1 GB and 10K scores included, 14-day retention	Pro $249/month, 5 GB and 50K scores, 30-day retention	Hybrid/on-prem on Enterprise	Teams where evaluation drives releases
Datadog LLM Observability	APM suite with LLM spans	Up to 40K LLM spans/month on the entry tier	100K LLM spans/month included, then $3.50 per 10K LLM spans (annual)	No	SRE teams already standardized on Datadog
OpenLLMetry + your backend	Vendor-neutral OpenTelemetry instrumentation	Open source	You pay your backend's ingest and retention	Yes	Teams with an existing observability stack

How to read this table: these LLM observability tools are priced on different units (Langfuse units, LangSmith base traces, Arize spans, Helicone requests, Braintrust GB and scores, Datadog LLM spans). A low entry price can hide expensive retention, content capture, evaluation runs, or seats, so price a realistic week of traffic instead of a demo.

Which LLM monitoring tool should you choose?

You want open source and might self-host later: Langfuse or Arize Phoenix.
You build on LangChain or LangGraph: LangSmith, because tracing works with minimal setup.
You want logs in an hour without touching app code: Helicone, or a gateway that already records every request.
Your releases are gated on evals: Braintrust or LangSmith.
SRE already owns Datadog: Datadog LLM Observability, so AI spans sit next to service traces.
You need vendor neutrality: instrument with OpenTelemetry following the GenAI semantic conventions, then send data to any backend.

What LLM observability tools need to explain

Traditional API monitoring tells you that a request returned 200, 429, or 500. That remains essential, but an LLM-backed feature can return 200 while producing invalid JSON, choosing the wrong tool, citing no source, exceeding the user’s patience budget, or requiring human rework.

LLM observability tools add context so engineers can separate those cases:

User report	Inspect first	Evidence that should be present
“The answer was wrong.”	Validation and evaluation	Prompt and retrieval version, output-schema result, tool result, sampled score
“The feature got slow.”	Critical path	End-to-end duration, time to first token, queue wait, retrieval and tool spans, attempt count
“Spend jumped.”	Usage and routing	Usage units, price-table version, retry amplification, fallback rate, accepted-task rate
“One provider is flaky.”	Route attempts	Provider request ID, status family, deployment alias, circuit-breaker state, fallback reason
“It worked yesterday.”	Release correlation	Model alias, route-policy version, prompt version, feature flag, deployment revision

Use OpenTelemetry semantic conventions where they fit so traces and metrics remain understandable across languages and backends. Treat the convention version as a schema dependency: GenAI conventions evolve, and an unplanned attribute migration can quietly break dashboards.

Score a tool before you instrument the whole product

Give each candidate 0–2 points for each question:

Can it preserve one trace across the API, queue, retrieval layer, tools, and model call?
Can an engineer query by request ID, model alias, route, prompt version, and validation result?
Can it ingest or export OpenTelemetry data without a proprietary rewrite?
Can it redact before persistence and enforce retention and role-based access controls?
Can it join online traces to offline evaluations and user feedback?
Can it separate estimated request cost from reconciled billing data?
Can it alert on a user-facing SLO, not only a provider error rate?
Can the team export raw data before a pricing or contract change becomes a migration incident?

No scorecard substitutes for a traffic sample. Run a seven-day trial on one representative workflow, record the actual retained data volume and people-hours, then decide.

Choose the access model before choosing a logo

Most LLM observability tools fall into one of five operating models.

Operating model	Choose it when	Typical integration	Costs people forget
OpenTelemetry to the existing backend	You already operate service telemetry and need one consistent trace estate	SDK or auto-instrumentation → collector → existing traces, logs, and metrics backend	Ingest, retention, high-cardinality fields, collector operations, and storage
AI-native tracing and evaluation platform	You need prompt/version workflows, trace review, and evaluation loops quickly	Vendor SDK, OTLP endpoint, or framework integration	Observation/event volume, trace retention, seats, evaluation runs, exports, and overages
APM suite with GenAI support	SRE already owns a single APM and wants AI context beside service telemetry	Existing agent plus GenAI instrumentation and custom attributes	Host/container pricing, custom metrics, log ingest, trace retention, and egress
Gateway ledger plus application telemetry	You route across providers and need route and cost facts at the application boundary	Gateway request ID or usage export joined to application traces	Gateway fees, export access, billing reconciliation, retention, and application join logic
Self-hosted tracing/evaluation stack	Data residency, customization, or predictable high-volume storage matters more than setup speed	SDK → collector → self-hosted tracing/evaluation components	Infrastructure, backups, upgrades, access controls, and the on-call rotation

The choice is not permanent, but the trace schema may be. Prefer an integration path that leaves the application’s correlation IDs and data contract under your control.

Integrate LLM observability tools around a user task

The root trace should represent a task that a user recognizes—not merely an HTTP call to a provider. Each material execution step becomes a child span.

user_task: create_support_reply
├── retrieve_account_context
├── assemble_prompt
├── model_route
│   ├── model_attempt: primary
│   └── model_attempt: fallback (only when needed)
├── validate_output
└── save_draft

Propagate the trace through HTTP, jobs, and tool calls. The W3C Trace Context recommendation defines portable traceparent and tracestate headers so context can survive service boundaries.

Start with a versioned, metadata-first contract

Store enough evidence to investigate production behavior while keeping raw content behind an explicit policy.

Field	Why it belongs in the contract	Do not put here by default
`trace_id`, `request_id`	Correlates the user task, logs, and support tickets	Raw user identity when a scoped surrogate works
`feature`, `route`, `environment`	Explains product and routing context	Unbounded URL, free-form prompt, full request object
`model_alias`, `provider`, `attempt`	Makes retry and fallback behavior visible	Credentials or authorization headers
`prompt_version`, `retrieval_version`	Connects change to regression	Raw prompt or retrieved document
`status`, `validation_result`, `error_class`	Separates transport success from usable output	Full stack trace in a metric label
`input_units`, `output_units`, `estimated_cost`	Supports near-real-time economics	A mutable price with no price-table version
`provider_request_id`	Helps escalation and support correlation	An ID exposed in a public error message

Keep high-cardinality data in traces or structured logs, not metric labels. Bounded metric dimensions such as feature, route, model alias, provider, environment, and result class are far easier to alert on.

A TypeScript pattern for attempts, validation, and fallback

The wrapper below stays independent of a particular model SDK. It records one outcome span and a child span for each routing attempt. Map standard fields to the OpenTelemetry GenAI semantic-convention version your team has pinned, and use a stable app.ai.* namespace for product-specific fields.

import { SpanStatusCode, trace } from "@opentelemetry/api";

const tracer = trace.getTracer("product-ai");

type Attempt = {
  provider: string;
  modelAlias: string;
  reason: "primary" | "retry" | "fallback";
};

type Result = {
  text: string;
  usage?: { input?: number; output?: number };
  providerRequestId?: string;
};

export async function runAnswer(
  attempts: Attempt[],
  invokeModel: (attempt: Attempt) => Promise<Result>,
  isValid: (result: Result) => boolean,
) {
  return tracer.startActiveSpan(
    "app.ai.request",
    { attributes: { "app.ai.feature": "support_reply", "app.ai.route": "default" } },
    async (requestSpan) => {
      try {
        for (const [index, attempt] of attempts.entries()) {
          const response = await tracer.startActiveSpan(
            "gen_ai.chat",
            {
              attributes: {
                "gen_ai.operation.name": "chat",
                "app.ai.provider": attempt.provider,
                "app.ai.model_alias": attempt.modelAlias,
                "app.ai.attempt": index + 1,
                "app.ai.reason": attempt.reason,
              },
            },
            async (attemptSpan) => {
              try {
                const result = await invokeModel(attempt);
                const valid = isValid(result);

                attemptSpan.setAttributes({
                  "app.ai.validation": valid ? "passed" : "failed",
                  "app.ai.input_units": result.usage?.input ?? 0,
                  "app.ai.output_units": result.usage?.output ?? 0,
                  "app.ai.provider_request_id": result.providerRequestId ?? "",
                });

                if (!valid) throw new Error("application_contract_failed");
                attemptSpan.setStatus({ code: SpanStatusCode.OK });
                return result;
              } catch (error) {
                attemptSpan.recordException(error as Error);
                attemptSpan.setStatus({ code: SpanStatusCode.ERROR });
                return null;
              } finally {
                attemptSpan.end();
              }
            },
          );

          if (response) {
            requestSpan.setAttribute("app.ai.selected_attempt", index + 1);
            requestSpan.setStatus({ code: SpanStatusCode.OK });
            return response;
          }
        }

        throw new Error("route_exhausted");
      } catch (error) {
        requestSpan.recordException(error as Error);
        requestSpan.setStatus({ code: SpanStatusCode.ERROR });
        throw error;
      } finally {
        requestSpan.end();
      }
    },
  );
}

The example deliberately does not attach raw prompts or completions. Some integrations capture content by default. For example, OpenLLMetry documents a setting to turn off prompt, completion, and embedding content tracing. Treat content capture as a redaction-aware, narrowly scoped debug or evaluation path—not a default span attribute.

Quick start: send one traced request with Python

The snippet below calls an OpenAI-compatible endpoint and records a span with model, latency, and token usage using the OpenTelemetry SDK. Swap the console exporter for an OTLP exporter pointing at Langfuse, Phoenix, Datadog, or your own collector.

# pip install openai opentelemetry-sdk
import os, time
from openai import OpenAI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-bot")

client = OpenAI(base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
                api_key=os.environ["LLM_API_KEY"])
model = os.environ.get("LLM_MODEL", "gpt-5.5")

with tracer.start_as_current_span("gen_ai.chat") as span:
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.request.model", model)
    t0 = time.time()
    r = client.chat.completions.create(model=model,
        messages=[{"role": "user", "content": "Summarize our refund policy in one line."}])
    span.set_attribute("gen_ai.usage.input_tokens", r.usage.prompt_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", r.usage.completion_tokens)
    span.set_attribute("app.ai.latency_ms", int((time.time() - t0) * 1000))
print(r.choices[0].message.content)

Price the operating model, not just the plan

To compare LLM observability tools honestly, calculate total monthly ownership for a representative workflow:

total observability cost =
  tool subscription or metered ingest
  + retained trace / log / metric storage
  + evaluation and annotation runs
  + data egress or export
  + self-hosted infrastructure
  + engineering and on-call time

Then relate it to business output:

effective cost per accepted task =
  reconciled model-and-tool cost / accepted tasks

This is more useful than cost per request. A low-cost model that causes retries, validation failures, or human correction can be the expensive choice. Use a model-price comparison workflow before changing production traffic based only on token price.

For cost views that infer spend from a model name and usage, retain both the estimated value and the authoritative ledger value. Langfuse, for example, documents that inferred cost depends on a model definition and usage data, while ingested cost can override it for custom agreements or exact provider values. That makes reconciliation a design requirement, not an afterthought. Read the Langfuse cost-tracking details.

Build three dashboards, then alert on outcomes

Do not begin with 40 panels. Start with three views that lead to distinct actions.

Dashboard	Question it answers	First measures
User outcome	Are users receiving valid results within the experience budget?	Validated success rate, p95 end-to-end latency, p95 time to first token, cancellations
Route and provider	Which decision or dependency changed?	Traffic share, `429`/`5xx`, retry amplification, fallback rate, route exhaustion, tool failure
Usage and economics	Which feature or change drives spend?	Units per accepted task, estimated vs. reconciled cost, retry cost, fallback delta, budget variance

Define validated success rate as:

eligible requests that pass the application contract / eligible requests

It is often a better availability signal than 2xx responses / requests, because it measures whether the product delivered what it promised.

Alert on user impact and use provider detail to diagnose it. The Google SRE Workbook’s alerting guidance makes the same core point: alerts should identify events that threaten an error budget, not every noisy symptom. A provider spike safely absorbed by a compatible fallback may need investigation, not a wake-up page.

A seven-day rollout for LLM observability tools

Day	Deliverable	Exit check
1	Root request ID, feature name, route, and trace propagation	A support ticket can find one complete trace
2	Model-attempt spans and output validators	Retry, fallback, and schema failure are distinguishable
3	Outcome and latency metrics	Validated success and p95 latency are visible by feature
4	Usage and cost estimate	The price-table version travels with every estimate
5	Redaction and retention controls	A sampled trace contains no unapproved secret or PII
6	Failure drill	Timeout, `429`, `5xx`, malformed-output, and tool-failure tests have expected evidence
7	SLO, alert owner, and incident note	A simulated user-impact alert reaches the right person with a trace link

For a deeper release gate, use the AI observability implementation checklist. Once the basics work, extend the operating view with AI routing API metrics and a clear AI API quota management policy.

Where a multi-model gateway fits

A gateway can centralize provider request IDs, usage records, routing decisions, and billing evidence. It does not replace application telemetry. Only the product knows the feature, prompt version, output contract, user-facing latency, and whether the task was accepted.

If you route several providers through Flatkey, pair the gateway usage ledger with the trace contract above. Keep a join key on both sides, reconcile gateway and provider usage with accepted-task outcome, and model fallback decisions as first-class events. Check Flatkey pricing for current commercial terms before making a cost assumption.

For log design details, read AI API observability logs.

For a cost-focused setup with code and provider cost APIs, see our LLM cost tracking guide.

FAQ

What are LLM observability tools?

LLM observability tools collect and join traces, metrics, logs, evaluations, and outcome data so a team can explain how an AI feature behaved in production. They should connect a user-visible result to the model, route, context, tools, latency, quality checks, and cost that produced it.

What are the best LLM observability tools in 2026?

For most teams the shortlist is Langfuse, LangSmith, Arize (Phoenix and AX), Helicone, Braintrust, and Datadog LLM Observability. Langfuse and Phoenix lead on open source, LangSmith on LangChain integration, Helicone on fast proxy-based setup, Braintrust on evaluation, and Datadog on joining AI spans with existing APM.

What is the difference between LLM monitoring and LLM observability?

LLM monitoring tracks known signals such as latency, error rate, token usage, and cost on dashboards and alerts. LLM observability goes further: it keeps enough trace context (prompts versions, retrieval, tool calls, validation results) to answer new questions about why a specific output was wrong or slow.

Which LLM observability tools are open source?

Langfuse, Arize Phoenix, Helicone, and OpenLLMetry publish open-source code that you can self-host. Self-hosting removes the vendor's usage fee but not the cost of storage, upgrades, access control, and on-call ownership.

Which LLM observability tools should a small team start with?

Start from the operating constraint, not the most familiar brand. A small team may prefer a managed free tier such as Langfuse Hobby, LangSmith Developer, or Helicone Hobby for fast trace review; an existing platform team may prefer OpenTelemetry instrumentation that reaches its current backend. Trial one representative workflow and price actual retention, evaluation, and operating work.

Should I log prompts and responses?

Not by default. Use metadata-first LLM observability, then allow narrowly sampled and redacted content capture only when a documented debugging or evaluation need justifies it. OpenTelemetry's security guidance recommends data minimization and reviewing what instrumentation emits.

Do LLM observability tools replace evaluations?

No. Observability tells you what happened to production traffic and helps you find samples to evaluate. Evaluations tell you whether behavior meets a rubric. The strongest production loop joins the two through a stable request, trace, or sample ID.

Start with one reliable question

The first success condition is modest: when a user reports a bad result, you can find the exact path, see whether the product contract passed, understand the routing decision, and estimate the cost of the attempt. Build that first. Once the data is trustworthy, LLM observability tools, dashboards, and optimizations become much easier to choose.