Observability in the Age of LLMs: A Field Guide

Last verified: 5 October 2026. Every product claim below carries a primary-source URL. Volatile figures (pricing, GitHub metrics, spec versions) are date-stamped and will drift.

Observability in the age of LLMs, defined

Observability in the age of LLMs is the practice of instrumenting a non-deterministic generative system so that every request is recoverable after the fact. Inputs, retrieved context, intermediate tool calls, model configuration and final output all have to be there, and they have to be gradeable rather than merely measurable. Classic observability answers “is the system up and fast?” This job is harder. It has to answer “was that answer any good, and why was it that answer?” There is no HTTP status code for that.

Three things break the old model. Output correctness is not a status code: a 200 OK can carry a fabricated citation. The same input yields different outputs, so a bug report containing only the user’s prompt is not a reproduction case. And cost moves with token count rather than request count, so a verbose model can double spend with traffic flat.

One note on phrasing before you search further. “In the age of LLMs” is a framing, not a product category. Tool documentation, pricing pages and job descriptions all use LLM observability. Search that term and you will find the implementations.

What actually changed: from three pillars to a graded trace

Observability in the age of LLMs still uses metrics, logs and traces. It cannot stop there. The old three were designed around four assumptions that generative systems violate.

Determinism. A reproduction case must now carry model ID with resolved version, temperature, top_p, seed, system prompt version and the exact retrieved context. Without those six fields you cannot re-run the failure.

Correctness. Quality has to be computed by an evaluator and written back onto the trace. Nothing in the transport layer emits it.

Cardinality. The dimensions that explain a regression are prompt version, tenant, user segment, tool name and model ID. Pre-aggregated metrics destroy exactly those. Average your latency across prompt versions and you have thrown away the only column that would have told you which prompt edit caused the slowdown. Wide, high-cardinality events are the correct storage shape. One fat row per request, queried arbitrarily afterwards. That is the substance of the “observability 2.0” argument Honeycomb has made in its own materials.

Cost. Spend is a telemetry field now, produced by tokens, retries and hidden intermediate agent steps. It is not a line item that arrives from procurement once a month.

Failures originate in five layers: the model and provider API, retrieval and embeddings, prompt and parameter logic, the orchestration or agent loop, and surrounding infrastructure. Honeycomb and Patronus each name four. Both omit orchestration, which is where agent systems now fail most often.

Here is the payload. A RAG assistant’s answer quality drops on a Tuesday. No code was deployed.

Candidate cause	Telemetry field that proves it
Retriever index rebuilt	Chunk IDs and similarity scores on the retrieval span, compared week over week
Provider silently rolled the model behind a floating alias	Resolved model version, not the alias string
Prompt edit shipped by another team	Prompt version hash on the LLM span
Temperature left high after an experiment	Model config attributes captured per request
Context window truncation dropped the relevant chunk	Prompt token count against model limit, plus a truncation event

Five causes, five distinct fields. Miss any one field and that cause becomes permanently indistinguishable from the others.

Observability vs monitoring vs evaluation vs guardrails

Four categories collide in this market, and most pages treat it as a two-way split.

Monitoring applies known thresholds to known signals and alerts you that something broke. You need it the moment anything is in production. Observability is arbitrary post-hoc interrogation of recorded events. It tells you why. You need it the first time an incident has no obvious cause. Evaluation scores output quality against criteria, offline against a dataset or online against live traffic. You need it before you can claim a change improved anything. Guardrails enforce inline, blocking or rewriting a request before the user sees it.

The practical difference nobody states: observability is read-only and after the fact. Guardrails sit in the request path and add latency to every single call. A tool that does both is making a latency trade on your behalf. Ask the vendor what it costs at p99.

Evaluation keeps being absorbed into observability products for a structural reason. An evaluator’s substrate is the full trace, meaning input, retrieved context and output. Whoever already stores traces is best positioned to score them. That is why TrueFoundry lists “LLM evaluation” as a pillar of observability in the first place (truefoundry.com).

A one-line test for each: if you can state the threshold in advance, it is monitoring. If you need to ask a question you did not anticipate, it is observability. If you need a score, it is evaluation. If you need to stop the output reaching a user, it is a guardrail.

Four pillars or five? Reconciling the competing frameworks

No standards body defines the pillars of observability in the age of LLMs. Every framework in circulation is vendor-authored, and they disagree on both the count and the categories. Treat each as a marketing artifact and attribute it accordingly.

Visual needed: Pillar framework reconciliation table comparing four published frameworks side by side Figure 1. One row per published framework. Columns for publisher, items named, what the framework omits, and whether each item is a signal, an activity or a process. Source the rows from the four vendor URLs below plus a neutral citation for the classic three pillars.

Framework	Published by	Items	Category problem
Five pillars	TrueFoundry (link)	LLM evaluation, traces and spans, retrieval/RAG, fine-tuning, prompt engineering	Mixes telemetry primitives with development activities
Four pillars	LaunchDarkly (link)	Data and prompt monitoring, model performance, user experience, risk and compliance	Organised by concern, not by data source; no instrumentation guidance follows from it
Four “pillars of effective observability”	Snorkel AI (link)	Model evaluation and testing, feedback loops, zero-/few-shot monitoring, interpretability	Different four entirely; interpretability is a research goal, not a collected signal
Classic three pillars	General software observability convention	Metrics, logs, traces (sometimes plus profiling or events)	Predates generative systems; has no slot for output quality

The five-pillar list has a category error worth naming. Fine-tuning is something you do. Prompt engineering is something you do. Traces and spans are something you collect. A reader who adopts that list as an instrumentation checklist will instrument three items and schedule two meetings.

Here is something defensible to use instead. It is this page’s own synthesis, offered as a proposed cut rather than an industry standard. Organise by where the data comes from, because provenance is what determines ownership, storage cost and privacy exposure:

1. Emitted telemetry. Spans, token counts, latency, errors. Owner: platform. Cheap. Low risk.
2. Captured content. Prompts, retrieved context, outputs, model config. Owner: application team. Expensive to store. Highest privacy exposure on the page.
3. Derived scores. Evaluator results, LLM-as-a-judge verdicts, schema validation. Owner: whoever owns quality. Cost scales with sampling rate and judge model price.
4. Human signal. Explicit feedback, edits, retries, escalations. Owner: product. Nearly free. It arrives late, and it is the only ground truth you get.

Each part has a different owner, a different storage cost profile and a different regulatory footprint. That is what makes the distinction operationally useful, and why it beats a list that puts fine-tuning next to spans.

What to instrument: a span-level reference

Observability in the age of LLMs lives or dies on span attributes. The vendor-neutral way to describe an LLM call is the OpenTelemetry GenAI semantic conventions. Before you write instrumentation against them, open that page and read the stability banner on each attribute group. Whether a group is stable or experimental decides whether code written this quarter survives a collector upgrade. Do not take a stability claim from any article, including this one, without checking the spec.

Fields a complete trace needs, grouped:

Group	Fields
Request identity	Trace ID, session ID, tenant, user segment
Model config	Provider, model ID resolved to version not alias, temperature, top_p, max_tokens, seed, tool schema hash
Content	System prompt version/hash, user input, retrieved chunk IDs and similarity scores, final output
Economics	Prompt tokens, completion tokens, cached and reasoning tokens where reported, retry count, computed cost
Timing	Queue time, time to first token, total duration per span
Outcome	Finish reason, schema validation result, evaluator scores, guardrail verdicts

Two fields get omitted constantly and regretted later. The first is the resolved model version behind a floating alias. Without it, a provider rollout is invisible and the resulting quality drop is undebuggable. The second is the prompt version identifier. Without it, you cannot attribute a quality change to a prompt edit. Both map directly to rows in the differential-diagnosis table above.

Span hierarchy for a RAG request: one root span per user request, with children for embed → retrieve → rerank → LLM call → post-process and validate. Retrieval spans must carry chunk IDs and similarity scores. Without them you can see that retrieval was slow. You can never see that it was wrong.

Visual needed: Annotated trace diagram of a RAG request from root span through validation Figure 2. Root span through embed, retrieve (chunk IDs, scores), rerank, LLM call (resolved model ID, temperature, prompt version, token counts, time to first token), schema validation, guardrail verdict, evaluator score. Label each span with the failure it diagnoses. The caption must state the semconv version the attribute names come from.

One trap on reasoning models. Intermediate reasoning tokens are billed and affect latency, but are often not surfaced in the response body. Cost telemetry derived from visible output length will under-count. Check how your specific provider reports these in its API reference before you trust any per-request cost figure.

Failure modes in the tools themselves: evidence from public issue trackers

This section is the one competitors cannot copy cheaply, because it requires reading primary artifacts rather than vendor prose. It also comes with a hard boundary. Everything below is a reported defect from a public tracker, not a measurement. Nothing here was reproduced, and severity is not characterised beyond what issue metadata supports.

The research task, stated so you can run it yourself and so a reviewer can check it:

Project	Issue tracker	What to search for
Langfuse	github.com/langfuse/langfuse/issues	Ingestion backpressure, self-host storage backends, SSO gating
Arize Phoenix	github.com/Arize-ai/phoenix/issues	Provider SDK bumps breaking instrumentation, streaming token counts
Helicone	github.com/Helicone/helicone/issues	Proxy-mode blind spots, cost attribution
OpenLLMetry	github.com/traceloop/openllmetry/issues	Instrumentation drift after provider releases
OTel semantic conventions	github.com/open-telemetry/semantic-conventions/issues	Filter to the gen-ai area for attribute renames

Visual needed: Failure-mode catalogue table, one row per reported defect class Figure 3. One row per reported defect class, with columns for symptom, affected tool and version, issue URL, open or closed, date opened, reaction or comment count, maintainer response, and implication for an evaluating team. Populate only from issues actually opened and read, and date-stamp the observation.

The recurring classes worth searching for, because they map to buying decisions rather than to bug-tracker noise:

• High-volume trace ingestion with dropped or queued spans. Implication: your sampling policy and the tool’s ingest path interact, and the trace you most need during an incident is the one most likely dropped.
• Self-hosted gaps versus managed. Storage backends, object storage configuration and SSO frequently differ. Implication: a self-host proof of concept may not represent the product you would buy.
• SDK breakage after a provider SDK bump. Implication: pin your instrumentation versions and read release notes on provider upgrades.
• Streaming responses producing incomplete or zero token counts. Implication: cost dashboards silently under-report for streaming endpoints.
• Cost attribution wrong for cached or reasoning tokens. Implication: the number on your observability dashboard and the number on your provider invoice are two different numbers until you reconcile them.
• PII in stored traces because redaction is opt-in. Implication: the default configuration is the risky one.

Primary artifacts show how fast this layer moves. TrueFoundry keeps a public changelog of its own gateway and guardrail releases (truefoundry.com/docs/changelog). Read the entries and their dates there. A vendor’s record of its own product is reliable about what shipped, and it is still the vendor’s wording.

The metrics worth tracking (and the ones that mislead)

Category	Metrics	The caveat nobody prints
System	P50/P90/P99 latency, time to first token, throughput, error rate split into infrastructure / business-logic / semantic	Mean latency is useless when response length varies by an order of magnitude; normalise per output token
Resource	CPU/GPU utilization, memory, token usage, cache hit rate	Near-meaningless if you consume hosted APIs. IBM and TrueFoundry both list CPU/GPU without this caveat; most readers have no access to those numbers
Quality	Correctness, faithfulness, groundedness, relevance, schema validity, refusal rate, toxicity flags	All are derived, all cost money to compute, none are emitted

Throughput deserves its own correction. On hosted APIs it is bound by your provider rate limit, not by your system’s capacity. You are measuring your quota.

Perplexity is presented as an LLM evaluation metric in at least one widely-circulated write-up (Medium, 16 November 2024). It requires token logprobs, which most hosted chat endpoints do not return, and it measures fluency rather than truth. A confidently fluent fabrication scores well. For production quality signal use faithfulness and groundedness scored against retrieved context, plus schema validity, which costs nothing and catches a real class of failure.

BLEU and ROUGE need a reference answer. Production traffic does not have one. They belong in offline regression testing against a version-controlled golden dataset, never on a live dashboard. G-Eval and other LLM-as-a-judge approaches work online precisely because they do not require a reference, which is also why they cost a model call each. The same holds for RAGAS-style RAG scorers and for evaluation runs logged through MLflow.

The signals most competitors omit entirely are behavioural: retry rate, edit-after-copy, conversation abandonment, escalation-to-human rate. No evaluator required, no extra inference cost, and they move before any judge model notices.

While we are correcting the record: the “750 million apps using LLMs by 2025” figure circulating from that same Medium post is unsourced on the page and is now a prediction about a year already past. Do not reuse it.

What observability itself costs

Zero competitors price observability in the age of LLMs. The cost is a function of four things: the billing unit, the stored payload size, retention beyond the included window, and online evaluator model calls.

Billing unit is the whole story. A platform billing per span charges an agent loop with twelve tool calls twelve times what it charges a single completion. A platform billing per trace charges the same for both. Record, from each vendor’s own pricing page with the date accessed:

Vendor	Pricing page	Record
Datadog LLM Observability	datadoghq.com/pricing	Billing unit, included retention
Langfuse	langfuse.com/pricing	Unit, retention, self-host tier limits
Arize	arize.com/pricing	Unit, seat structure
LangSmith	langchain.com/pricing	Unit, seat structure
Helicone	helicone.ai/pricing	Unit, retention

Read the per-unit rates off those five pages yourself and date-stamp what you read. No secondary summary is current enough to trust.

Build the model as arithmetic, with every input labelled an assumption:

• Assumption A: requests per month (model three tiers: 10,000 / 250,000 / 2,000,000).
• Assumption B: spans per request, roughly 5 for a RAG pipeline and roughly 15 for an agent loop.
• Assumption C: average stored payload per span, derived from prompt, retrieved context and output token counts at approximately 4 bytes per token.
• Assumption D: sampling rate for storage, and separately for evaluation.
• Assumption E: judge model price per million tokens, from the provider’s own pricing page.

Monthly cost ≈ (A × B × sampling) × vendor unit rate + stored bytes × storage rate + (A × eval sampling × judge tokens × E).

Visual needed: Stacked cost chart of monthly observability spend at three request volumes Figure 4. Monthly spend at 10,000, 250,000 and 2,000,000 requests per month, split into trace ingestion, payload storage, extended retention and online evaluator calls, with RAG (≈5 spans) and agent (≈15 spans) series side by side. The caption must state that this is a derived model from published rates rather than a measurement, and list every assumption.

The evaluator term is the one that surprises people. Online evaluation means a second model call per sampled trace. At 100% sampling with a capable judge model, evaluation can approach the cost of the application itself. Halving the sample halves that line exactly. There is no fixed component to amortise.

Self-hosting trades licence spend for storage, retention management and on-call. Read the licence file rather than the marketing page. Start with Langfuse’s LICENSE and each project’s open-source-versus-commercial documentation. Several projects in this category gate SSO, fine-grained RBAC or data-access controls behind a commercial tier. “Open source” and “full feature set self-hosted” are not the same claim.

The tooling landscape: what each category is actually for

Sort the market for observability in the age of LLMs into four categories and you stop comparing things that are not comparable.

Category	Examples	What you get	What it cannot see
LLM-native platforms	Langfuse, Arize Phoenix, Arize AX, LangSmith, Braintrust, Comet Opik, Weights & Biases Weave, TruLens	Prompt versioning, evaluators, text-first trace UI	Your non-LLM services, unless you also send those
APM with an LLM module	Datadog LLM Observability, Honeycomb, Grafana with Tempo and OTel, New Relic	One pane with the rest of your stack	Prompt management and evaluator tooling, mostly
Gateways and proxies	Helicone, TrueFoundry, LiteLLM, Portkey	Telemetry with near-zero code change	In-process retrieval, embedding and tool steps. The gateway only sees what crosses it
Evaluation-first	Patronus AI (and Percival), Snorkel AI, OpenAI Evals, DeepEval, RAGAS	Scoring depth, judge tooling	Infrastructure-layer telemetry

Visual needed: Primary-source tool matrix with one row per tool and sourced cells Figure 5. Rows for Langfuse, Arize Phoenix, LangSmith, Datadog LLM Observability, Helicone, OpenLLMetry with Grafana and OTel, Patronus and TrueFoundry. Columns for category, SPDX licence from the LICENSE file, self-host availability, features gated commercially, OTLP ingest and export, GenAI semconv conformance, billing unit, latest release date and open issue count. Footnote every cell with a source URL and date-stamp the caption.

The gateway blind spot deserves emphasis because it is the most common disappointment in this category. A proxy gives you token counts and latency on day one with a base-URL change. It cannot show you which chunk the retriever returned, because retrieval never crossed the proxy.

Patronus publishes a tool comparison table with the right shape (patronus.ai). It carries no licences, no prices, no versions and no observation dates, and it rates Patronus best or tied-best in every column. It asserts that Arize Phoenix, LangSmith and Helicone lack dedicated PII scanners, and that LangSmith only integrates with LangChain. Those are vendor claims about rivals. Verify each against the rival’s current documentation before repeating them anywhere, and label them as marketing if you do.

The Grafana plus OpenTelemetry path, which no competitor covers: you can run LLM observability on OTel instrumentation (OpenLLMetry from Traceloop, or OpenInference) into a trace backend you already own, whether that is Tempo, Jaeger or ClickHouse, alongside Prometheus metrics. Free: traces, latency, token attributes, one unified view with your existing services. You build the evaluators, the prompt-version management, and a UI for long text payloads. That last item is underrated. Trace viewers designed for microsecond spans are genuinely bad at displaying a 4,000-token prompt next to a 2,000-token retrieved context.

Visual needed: Decision tree answering which observability layer a team needs Figure 6. “Which layer do I need?” branching on hosted API versus self-hosted model, RAG versus agent, regulated data yes or no, existing APM yes or no, and open-source-only constraint. Each leaf terminates in one of the four categories with its trade-off named.

Instrumenting a RAG pipeline: the minimum viable setup

Day one, in order:

1. Install OTel-based auto-instrumentation for your provider SDK.
2. Create a root span per user request carrying tenant and session attributes.
3. Wrap retrieval in its own span with chunk IDs and similarity scores.
4. Attach the prompt version identifier to the LLM span.
5. Read token counts from the provider’s usage field. Do not estimate them.
6. Emit one structured outcome event with finish reason and schema-validation result.

Write the spans with the OpenTelemetry API for your language. You need a tracer, a started span, attribute setters and an ended span. Check the exact function signatures against the current OpenTelemetry documentation rather than against any article, including this one. The APIs differ between Python and TypeScript, and both have moved.

A staged rollout that works because it puts the expensive part last:

Week	Add	Why here
1	Traces and token counts	Nothing else is interpretable without them
2	Structured outcome events, schema validation	Free quality signal, no model calls
3	Sampled offline evaluator against a small golden dataset	Catches regressions before they ship
4	Online evaluation on a sampled slice, with a hard budget cap	Expensive, and worthless until you can slice by prompt version

Four day-one mistakes: logging raw prompts before redaction exists, shipping with no sampling policy so the bill arrives before the insight, instrumenting the LLM call but not retrieval, and grouping every dashboard by model alias instead of resolved version.

Sampling, retention and keeping PII out of your traces

Four sampling strategies, each right in a different place. Head or random sampling controls cost on healthy traffic. Tail sampling decides after the trace completes, so it can keep every error and every slow request. Those are the ones you actually need. Rule-based sampling by tenant or feature flag covers the customer who is about to escalate. Trigger-based capture records in full when an evaluator scores low or a guardrail fires.

The production combination most teams converge on: 100% of errors and low-scoring traces, a small percentage of everything else.

Redaction can happen in three places. In the SDK before export, in an OpenTelemetry collector processor, or at the vendor’s ingest. Only the first two keep regulated data out of a third party’s storage at all. Vendor-side redaction means the raw data crossed the boundary before it was scrubbed. That distinction matters for a data processing agreement, for SOC 2 scope, and for any data loss prevention control you already run.

There is a structural tension nobody states. Observability in the age of LLMs stores text, and the fields that make a trace debuggable are exactly the fields a retention policy wants to delete. Full prompt, full retrieved context, full output. The resolution teams actually use: keep hashes, token counts, scores and model config long-term, and keep full payloads for days rather than months. What you lose is specific. You can still see that prompt version 14 regressed. You can no longer read the bad answer.

If traces are the audit log for a regulated workload, retention is set by the regulation, not by your debugging convenience. Obligations under the EU AI Act and sector rules vary by role and risk class. Take that to your compliance owner and the primary text, not to a blog post.

Agents break the trace: observability for multi-step systems

Agents are where observability in the age of LLMs stops being a tracing problem. An agent loop has a variable, model-decided number of steps. Span count per request is unbounded. So is cost per request. And the failure is usually a wrong decision rather than a wrong output: the model picked the wrong tool, looped on the same call, or quietly gave up and summarised its own failure as an answer.

The signals that matter here barely overlap with single-call metrics. Steps per task. Tool-call success rate broken out per tool. Loop and repetition detection, context window utilisation with truncation events, task completion rate, and cost per completed task. Cost per call is the wrong denominator for agents. An agent that halves its per-call cost by taking four times as many steps looks like an improvement and is not.

Every tool call is an external dependency with its own latency and failure mode. MCP servers add a hop that most LLM-native tracing misses unless you instrument it deliberately. Gateway-layer MCP policy is one direction the industry is taking, and vendor changelogs are where you will see it land first.

Multi-turn sessions need a layer above traces. A conversation that degrades over five turns has no single bad trace in it. You need session-level grouping and a session-level score, which most tooling still treats as an afterthought.

A rollout checklist

• Resolved model version recorded as a span attribute, not the floating alias
• Prompt version identifier attached to every request
• Token counts read from the provider usage field, never estimated
• Retrieval spans carry chunk IDs and similarity scores
• Redaction runs in the SDK or collector, before export
• Tail sampling keeps 100% of errors and low-scoring traces
• Evaluator spend has a hard monthly cap, enforced not documented
• At least one dashboard sliced by prompt version
• Alert thresholds set on P95/P99, not means
• Retention policy written down, with a named owner
• A golden dataset exists and is version-controlled
• A rollback path for a prompt change exists and has been exercised
• Streaming endpoints verified to report non-zero token counts
• Cached and reasoning tokens reconciled against the provider invoice once
• Agent dashboards use cost per completed task, not cost per call

Two questions for any vendor, both derived from the research above. What is your billing unit, and how does it count an agent loop with twelve tool calls? Which features in your documentation are gated behind a commercial licence?

What this page deliberately does not claim: no overhead or latency benchmarks appear anywhere in it. SDK instrumentation overhead at p99, and the accuracy of vendor cost attribution for cached and reasoning tokens, are genuinely unresolved in public sources. Answering either requires running the tooling against your own traffic, and the answer is workload-specific. Measure it on a mirrored slice of production before you standardise.

Frequently Asked Questions

What does LLM observability mean?

LLM observability is the practice of instrumenting an LLM-powered application so every request’s inputs, retrieved context, model configuration, intermediate tool calls and output are recorded and can be interrogated and scored afterwards. It adds output quality as a first-class signal, because a successful HTTP 200 response can contain a completely fabricated answer.

What are the five pillars of LLM observability?

The five pillars, as published by TrueFoundry, are LLM evaluation, traces and spans, retrieval/RAG, fine-tuning and prompt engineering. This is a vendor-authored framework, not a standard: LaunchDarkly publishes a different four, Snorkel AI publishes another four. See the reconciliation table above for all four side by side.

What is observability in AI systems?

Observability in AI systems is the ability to understand a system’s behaviour from the telemetry it emits, spanning data and features, model version and configuration, inference performance, and output quality. The LLM-specific delta: open-ended text outputs have no ground truth, so quality must be derived by evaluators rather than read off a status code.

What are the four pillars of observability?

Two different questions hide here. Classic software observability is usually described as three pillars, meaning metrics, logs and traces, sometimes extended to four with profiling or events. Separately, LaunchDarkly and Snorkel AI each publish a four-pillar framework specific to LLMs, and the two lists do not match.

Is LLM observability different from LLM monitoring?

Yes. Monitoring alerts on thresholds you set in advance. Observability lets you ask questions you did not anticipate by querying recorded high-cardinality events. You need both, and LLM systems need a third thing as well: evaluation, because neither monitoring nor tracing scores whether the output was actually any good.

Can I do LLM observability with OpenTelemetry alone?

Yes, with caveats. The OpenTelemetry GenAI semantic conventions plus an instrumentation library give you traces, latency and token attributes in any backend you already run. You build evaluators, prompt-version management and a text-friendly UI yourself. Check the spec’s stability status first. Attribute groups carry their own status and can change between releases.

How much does LLM observability cost?

Cost is driven by four things: billing unit (trace, span, ingested GB or seat), stored payload size, retention beyond the included window, and online evaluator model calls. Per-span billing makes agent loops dramatically more expensive than single completions. Use the cost model above and substitute your own volumes and sampling rates.

Which LLM observability tools are open source?

Langfuse, Arize Phoenix, Helicone, OpenLLMetry and DeepEval all publish source code. Check each project’s LICENSE file for the exact SPDX identifier on the date you evaluate. Several gate SSO, fine-grained RBAC or data-access controls behind a commercial tier, so open source does not mean the full feature set self-hosted.

---

Further reading: Langfuse vs LangSmith comparison · LLM observability tools compared · OpenTelemetry GenAI reference · LLM evaluation metrics explained · LLM-as-a-judge costs · Building golden datasets · Agent observability and MCP tracing · Datadog LLM Observability pricing · Self-hosting Langfuse · PII redaction in LLM telemetry · Prompt versioning and rollback · RAG failure modes

Changelog. 5 October 2026: first publication. Pricing table, tool matrix and failure-mode catalogue require live verification before publication and re-verification quarterly. The semconv version must be stated once confirmed against the spec page.

The practical test of whether observability in the age of LLMs is working at your organisation is narrow and checkable. Pick last week’s worst answer. See whether you can recover the resolved model version, the prompt version, the retrieved chunks and the token counts that produced it. If any one of those four is missing from the trace, that is the next thing to instrument.