Reliability and RoutingJune 18, 2026Flatkey Team

AI Observability Platforms: What They Do and How to Choose (2026)

AI observability platforms record prompts, tokens, cost, agent steps, and quality scores. Compare Langfuse, LangSmith, Arize, Datadog, Helicone, Braintrust, and Logfire on features and October 2026 pricing, with OpenTelemetry code.

AI Observability Platforms: What They Do and How to Choose (2026)

An AI observability platform records every model call, agent step, and tool call in your application (prompt, response, tokens, cost, latency, and errors) so you can debug, evaluate, and control spend in production. The leading options in October 2026 are Langfuse, LangSmith, Arize (Phoenix and AX), Datadog LLM Observability, Helicone, Braintrust, and Pydantic Logfire, and most of them accept OpenTelemetry traces. Choose based on where your data must live, how you evaluate quality, and what you will pay per trace.

This guide explains what AI observability is, which signals to capture, how the main platforms compare on features and published pricing, and how to instrument an app in under 30 lines of code. Prices are taken from each vendor's official pricing page as of October 24, 2026.

What is AI observability?

AI observability is the practice of collecting traces, metrics, logs, and quality scores from AI systems so you can answer three questions at any time: What did the model do? Was the answer good? What did it cost? It extends classic application monitoring, which tracks uptime and latency, with signals that only matter for language models:

Inputs and outputs: the prompt, retrieved context, and the generated answer.
Token usage and cost: input, output, and cached tokens per request, user, and feature.
Agent traces: each reasoning step, tool call, and retry in a multi-step run.
Quality: evaluator scores, user feedback, and hallucination or policy flags.
Model and provider metadata: which model answered, through which route, with which parameters.

Traditional monitoring tells you a request took 9 seconds. AI observability tells you it took 9 seconds because the agent called a search tool three times, the second model call hit a 429, and the fallback model produced an answer that failed your groundedness check.

Monitoring vs observability vs evaluation

Practice	Question it answers	Typical output
Monitoring	Is the system up and within thresholds?	Dashboards, alerts on latency and error rate
Observability	Why did this specific request behave this way?	Traces with prompts, tool calls, tokens, metadata
Evaluation	Is the output correct, safe, and useful?	Scores from tests, LLM judges, or human review

A modern AI observability platform combines all three. The NIST AI Risk Management Framework also stresses ongoing measurement and monitoring of deployed AI systems, which is why many governance teams now ask for these records.

The open standard: OpenTelemetry GenAI conventions

You no longer need a vendor-specific SDK to capture AI telemetry. OpenTelemetry defines semantic conventions for generative AI that name the standard attributes for model calls. As of October 2026 the conventions are maintained in a dedicated semantic-conventions-genai repository and are still marked "Development," so expect some names to change.

Attribute	Meaning
`gen_ai.operation.name`	The operation, such as `chat` or `embeddings` (required)
`gen_ai.provider.name`	The provider, such as `openai` or `anthropic` (required)
`gen_ai.request.model`	Model requested by your code
`gen_ai.response.model`	Model that actually answered
`gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens`	Token counts
`gen_ai.response.finish_reasons`	Why generation stopped
`gen_ai.conversation.id`	Groups turns in one conversation

Instrument with these attributes once and you can send the same traces to Langfuse, Arize Phoenix, Datadog, Logfire, Grafana Tempo, or any other OTLP-compatible backend. Context propagation across services follows the W3C Trace Context standard.

AI observability platforms compared

Platform	Deployment	Strengths	Published entry pricing (Oct 2026)
Langfuse	Open source (self-host) or cloud	Tracing, prompt management, evals; MIT-licensed core	Hobby free (50k units/month); Core $29/month; Pro $199/month
LangSmith	Cloud, hybrid, or self-hosted on Enterprise	Deep LangChain and LangGraph integration, evals	Developer $0/seat (5k base traces); Plus $39/seat
Arize Phoenix and AX	Phoenix open source; AX cloud	OTel-native tracing, evals, drift analysis	Phoenix free to self-host; AX Pro $50/month (50k spans)
Datadog LLM Observability	Cloud (part of Datadog)	Joins LLM traces with APM, logs, and infrastructure	Usage-based; quote via Datadog
Helicone	Cloud or self-host	Proxy-based logging with one line of config, caching	Hobby free (10k requests); Pro $79/month; Team $799/month
Braintrust	Cloud	Evaluation-first workflow, experiments, scoring	Starter $0 (1 GB processed, 10k scores); Pro $249/month
Pydantic Logfire	Cloud	General OTel observability with strong Python and agent tracing	Personal free (10M records/month); Team $49/month

How to read this table:

Need to keep data in your own environment? Start with Langfuse or Arize Phoenix, both of which you can self-host.
Already pay for Datadog? Its LLM Observability product keeps AI traces next to your existing APM and logs.
Built on LangChain or LangGraph? LangSmith requires the least setup.
Evaluation is the priority? Braintrust and LangSmith lead with experiments and scoring workflows.
Want zero code changes? Proxy-style tools like Helicone, or the logs from your AI gateway, capture traffic at the network layer.

What to look for in an AI observability platform

Use this checklist when you run a trial:

OpenTelemetry support. Can it ingest OTLP traces with gen_ai.* attributes, and can you export your data back out?
Agent and tool tracing. Does it show nested spans for every step, tool call, and retry, not just the final model call?
Cost tracking. Does it calculate cost per request from tokens and your real prices, including cached tokens and batch discounts?
Evaluations. Can you run automated checks (LLM-as-judge, regex, reference answers) on production traces and alert on score drops?
Privacy controls. Can you redact or hash prompts, set retention periods, and restrict who sees raw content?
Pricing unit. Vendors bill per trace, span, "unit," GB, or seat. Estimate your monthly volume in each unit before comparing.
Self-hosting. If regulated data is involved, check that a self-hosted option exists and what it costs to run.

Instrument an LLM call in Python with OpenTelemetry

This example records one chat completion with the standard GenAI attributes and exports it to any OTLP endpoint (Langfuse, Phoenix, Logfire, or a local collector). It calls models through an OpenAI-compatible endpoint, so the same code works for GPT, Claude, Gemini, or DeepSeek.

pip install openai opentelemetry-sdk opentelemetry-exporter-otlp
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"   # your collector or vendor endpoint
export FLATKEY_API_KEY="sk-..."

import os, time
from openai import OpenAI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-bot")

client = OpenAI(api_key=os.environ["FLATKEY_API_KEY"],
                base_url="https://router.flatkey.ai/v1")

def answer(question: str, model: str = "claude-sonnet-5-5") -> str:
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "flatkey")
        span.set_attribute("gen_ai.request.model", model)
        t0 = time.time()
        r = client.chat.completions.create(
            model=model, max_tokens=300,
            messages=[{"role": "user", "content": question}])
        span.set_attribute("gen_ai.response.model", r.model)
        span.set_attribute("gen_ai.usage.input_tokens", r.usage.prompt_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", r.usage.completion_tokens)
        span.set_attribute("gen_ai.response.finish_reasons", [r.choices[0].finish_reason])
        span.set_attribute("app.latency_ms", int((time.time() - t0) * 1000))
        return r.choices[0].message.content

print(answer("How do I reset my password?"))
provider.shutdown()  # flush spans before exit

In Node.js the pattern is the same with @opentelemetry/sdk-node:

import OpenAI from "openai";
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("support-bot");
const client = new OpenAI({ apiKey: process.env.FLATKEY_API_KEY, baseURL: "https://router.flatkey.ai/v1" });

export async function answer(question, model = "gpt-6.1-sol") {
  return tracer.startActiveSpan(`chat ${model}`, async (span) => {
    span.setAttributes({ "gen_ai.operation.name": "chat", "gen_ai.request.model": model });
    const r = await client.chat.completions.create({ model, messages: [{ role: "user", content: question }] });
    span.setAttributes({
      "gen_ai.response.model": r.model,
      "gen_ai.usage.input_tokens": r.usage.prompt_tokens,
      "gen_ai.usage.output_tokens": r.usage.completion_tokens,
    });
    span.end();
    return r.choices[0].message.content;
  });
}

Keep prompt and response bodies out of span attributes unless your privacy policy allows it; many teams log a hash or a redacted copy instead. Our payload logging guide covers redaction patterns.

Your AI gateway is an observability source too

If your app calls models through an AI gateway or router, the gateway already sees every request, so it is the cheapest place to start. It captures data even for services that have not been instrumented yet. Flatkey's usage logs, for example, record the timestamp, model, input tokens, output tokens, cost, masked API key, and status for every call, can be exported as CSV or JSON, and support monthly spend caps and per-group quotas (Flatkey usage docs, October 2026).

A practical setup combines both layers:

Layer	What it sees best	Example use
Gateway logs	Every model call, cost, status, provider errors, fallbacks	Spend by key or team, 429 and 5xx rates, provider outages
Application traces	Business context, agent steps, retrieval, tool calls	Why one user's request failed or answered badly
Evaluations	Answer quality over time	Catching regressions after a prompt or model change

Join the two with a shared request ID: send your trace ID as a header or metadata field on every model call so you can move from a gateway log row to the full application trace. See LLM API observability metrics, traces, and cost for the metric set we recommend, and the AI gateway KPI dashboard for chart layouts.

The metrics that matter in production

Metric	Why it matters	Alert idea
p50 / p95 latency per model	Users feel tail latency	p95 above your SLO for 10 minutes
Time to first token	Streaming experience	Rising TTFT on a single provider
Error rate by status (429, 5xx, timeout)	Provider or quota problems	429 rate above 2%
Fallback rate	Hidden reliability issues	Fallbacks doubling day over day
Cost per request and per user	Budget control	Daily spend above forecast
Tokens per request	Prompt bloat and runaway agents	Agent runs above a token ceiling
Evaluation score	Quality regressions	Score drop after deploy

Attribute cost by team and environment so finance questions have answers; the cost attribution guide and cost tracking by environment show tagging schemes. For a rollout plan, follow the AI observability implementation checklist.

FAQ

What is AI observability?

AI observability is collecting traces, metrics, logs, and quality scores from AI applications so you can see what a model or agent did, whether the result was good, and what it cost. It adds prompts, tokens, tool calls, and evaluations to traditional monitoring.

What is the best AI observability platform?

There is no single best platform. Langfuse and Arize Phoenix suit teams that want open source and self-hosting, LangSmith suits LangChain users, Datadog suits teams already on Datadog, and Braintrust suits evaluation-heavy workflows. Trial two options with your own traces.

Is there a free AI observability platform?

Yes. As of October 2026, Langfuse Hobby (50k units per month), LangSmith Developer (5k base traces), Helicone Hobby (10k requests), Braintrust Starter, and Logfire Personal (10M records) are free, and Langfuse and Arize Phoenix can be self-hosted at no license cost.

What is the difference between LLM monitoring and AI observability?

Monitoring watches predefined thresholds such as latency and error rate. Observability lets you investigate any individual request in detail, including prompts, tool calls, token usage, and quality scores.

Does AI observability work with OpenTelemetry?

Yes. OpenTelemetry's GenAI semantic conventions define standard attributes such as gen_ai.request.model and gen_ai.usage.input_tokens, and most platforms accept OTLP traces. The conventions are still in Development status as of October 2026.

What is generative AI observability?

It is the same practice applied to generative models: capturing prompts, outputs, token costs, agent steps, and evaluation results so teams can debug and improve generative AI features in production.