10 Observability Signals for Multi-Step LLM Systems

A support agent classifies the intent correctly, retrieves three chunks, calls the refund tool with a valid JSON payload, and returns a confident answer in 4.2 seconds. Every span returns 200. The chunks were for a different product line. The refund tool got the right schema with the wrong order ID. The customer was told they’d been refunded when they hadn’t.

Nothing in the error dashboard moved.

That is the gap observability in multi-step LLM systems has to close, and more logging does not close it. What closes it is one causally linked trace that survives async boundaries, streaming responses, sub-agent handoffs and retries, with quality, cost and version data attached to each node.

This page gives you ten signals a multi-step trace must carry, an annotated example trace, a failure catalogue built from public GitHub issues with reaction counts, a cost model derived from published rate cards, and a tool-to-signal mapping.

Disclosure: this page sells no observability product. Four of the pages currently ranking for this query are vendor listicles that place their own product at #1: Confident AI, Braintrust, Galileo and OpenObserve. Every product weakness cited below links to a public issue, changelog entry or repository field. All repository facts, issue statuses and list prices in this article were checked on 8 August 2026 and are re-verified quarterly.

---

What observability actually means in an LLM system

LLM observability is the practice of capturing every input, intermediate step and output of an AI application as one linked trace, with quality, cost and latency measurements attached to each step.

Four object types carry the whole model. Vendors use these words inconsistently, so pin them down before you compare products:

Object	Definition	Example
Span	One unit of work	A single LLM call, one vector search, one tool invocation
Trace	The full tree of spans for one user request	The ten-step agent run that answered “where’s my refund?”
Session / thread	Multiple traces from one conversation	Six turns of back-and-forth with the same customer
Score	A numeric or categorical judgement attached to a span, trace or session	faithfulness: 0.41 on the generation span

Four pillars follow. Tracing is the linked span tree. Evaluation is scores at the right granularity. Cost and usage metering is tokens and spend per span. Version lineage is the prompt, model snapshot and code version stamped on every span. OpenObserve’s guide cites a “CHI 2025 study with 30 developers” framing observability as awareness, monitoring, intervention and operability. No paper title, authors or DOI is published, so treat that framing as unverified until someone locates it in the ACM Digital Library [NEEDS SOURCE].

One vocabulary trap. The current #1 result uses “traceability” in a sense that reads like audit logging. In a span-tree system it means causal linkage between spans, so you can answer which step caused this. That same page expands RAG as “Relevance-Aware Generative”. It is retrieval-augmented generation.

---

Why observability in multi-step LLM systems breaks both APM and single-call logging

Three regimes, not two. Most comparison articles contrast APM with LLM logging and stop, which is why they never describe what actually goes wrong inside an agent.

	Traditional APM	Single-call LLM logging	Multi-step observability
Primary question	Did the system respond?	What did the model say?	Which of the ten steps went wrong?
Core signals	Uptime, p99 latency, error rate	Prompt, completion, tokens, cost	Span tree, tool arguments, retrieval provenance, handoff context, termination reason
Unit of analysis	Request	Call	Trace
Failure it catches	500s, timeouts, saturation	Bad output on a known prompt	Silent partial success
Blind spot	Semantics	Everything upstream and downstream of the call	Nothing, if context propagation holds

Error masking. Chains with retries and fallbacks swallow failures by design. The retry succeeds, the fallback returns something, the trace records success. Galileo’s framing, that agent failures rarely surface as clean exceptions, is the correct instinct. Made concrete: a reranker that times out and falls back to raw vector order produces a fully green trace and a materially worse answer.

Latency compounding. Take a six-step chain where each step has a p50 of 400ms and a p99 of 3s. The p50 lands near 2.4s. If one step hits its p99 while the other five stay at p50, the user waits 5s. If two do, 7.6s. The tail is dominated by whichever step has the fattest distribution, and an end-to-end average never tells you which. Per-step histograms do.

Fan-out economics. Count the judge calls, the reflection loop, the retries and the guardrail passes, and a single request can be 8 to 30 model calls. Per-request cost dashboards understate spend. Per-call dashboards can’t be attributed to a feature or a customer.

Non-determinism. A bug report without the model snapshot, temperature, seed, the assembled prompt and the retrieved chunk IDs is not reproducible. You will re-run it, get a different answer, and close the ticket.

---

The 10 signals of observability in multi-step LLM systems

Use this as an audit checklist against your current instrumentation. Most teams have signals 1 through 3 and none of 4 through 10.

The ordering is not arbitrary. Signals 1 to 3 are structural, and you cannot add them later without re-instrumenting, because they change how spans are created. Signals 4 to 7 are step-semantic, describing what a step did, and can be added incrementally. Signals 8 to 10 are the improvement loop, turning production traffic into prioritised work.

Visual needed: Hero diagram, the span tree of a 10-step support agent (guardrail-in, intent classification, query rewrite, embed, vector search, rerank, tool call, generation, guardrail-out, feedback attachment), with each of the 10 signals labelled onto the span it attaches to. Data: the worked example step list below, plus OpenTelemetry GenAI semantic convention attribute names verified against the current spec. No product screenshots.

---

1. Unbroken parent-child trace context across the whole chain

Every span carries a trace ID and a parent span ID. That is the entire mechanism, and it fails constantly. A broken parent link doesn’t throw. It produces orphan traces that render as separate user requests, so span-count metrics look fine and debugging becomes impossible.

Three environments break propagation reliably:

• Async execution. Implicit context (Python ContextVars, Node async hooks) does not always survive a task boundary, a thread pool hop, or a library that schedules work itself.
• Streaming responses. The generator yields, the framework returns control, and the context that was active when the span opened is gone by the second chunk.
• Serverless and short-lived processes. The function returns, the runtime freezes, the exporter never flushes. The spans existed. They just never left.

The first two are documented. Langfuse issue #3961 reports that FastAPI StreamingResponse resets ContextVars after the first yielded chunk, producing multiple traces for one request, with 49 reactions. Issue #8780 reports “Failed to detach context” raised repeatedly with the LangChain CallbackHandler on async methods such as ainvoke, which also covers LangGraph, with 62 reactions.

Mitigation, in order of value:

1. Propagate context explicitly across task boundaries. Pass the parent span in rather than trusting an ambient context var.
2. Force a flush before process exit in Lambda, Cloud Run, Vercel functions, and any container that can be reaped mid-request.
3. Add a synthetic canary trace in CI that asserts span count and tree depth. A propagation regression then fails the build instead of quietly degrading a month of production data.

That third one is the highest-leverage practice in this article. It appears in none of the ranking pages.

Visual needed: Before/after diagram, the same request rendered as one 10-span tree versus three orphaned fragments after an async context break, annotated with the exact boundary where propagation is lost. Data: the failure descriptions in Langfuse issues #3961 and #8780. Illustrative diagram only, not a captured screenshot.

---

2. Per-step latency and the critical path

Record four timings per span, because they have completely different fixes:

• Queue or wait time. Points at concurrency limits or provider rate limiting.
• Time-to-first-token. What the user perceives. Points at prompt size, cold routing or a slow prefix.
• Total generation time. Points at output length and model choice.
• Retrieval sub-timings. Embed, vector search and rerank recorded separately. A slow embed is a caching problem. A slow search is an index problem. A slow rerank is a top-k problem.

Critical path is not total. With parallel retrieval, the sum of span durations exceeds wall-clock time by a wide margin. In the worked example below, ten spans sum to 3,540ms while the request completes in roughly 3.1 seconds, because embed, search and rerank overlap with the order lookup. Teams that sort spans by duration and optimise the longest one routinely find the chain got no faster.

Fixes are specific. Cache embeddings for repeated queries. Cut top-k so the reranker does less work. Move reranking off the critical path where ordering only affects a secondary display. Stream the first token before post-processing finishes.

---

3. Token and cost attribution at span level, rolled up to cost-per-completed-task

Record on every LLM span: prompt tokens, completion tokens, cached and read tokens, reasoning tokens where the provider exposes them, the model ID, and the unit price at time of call.

That last one matters more than it sounds. Rate cards change. A tool that computes cost by joining against a price table later will silently reprice your history.

Cost-per-completed-task is the metric for finance. Total spend across every span in the trace, divided by the number of traces that reached a successful terminal state. Abandoned, timed-out and guardrail-blocked traces all consumed tokens. Naive dashboards exclude them. At an 18% failure rate your real cost per completed task is 22% above what the dashboard says; at 30%, 43% above.

Price tables go stale in public. Helicone issue #2983 is a user filing a request purely to get the gpt-4o-2024-11-20 price updated. Record the unit price on the span and you are immune. Our pricing LLM features guide works the calculation through a tiered product.

One billing wrinkle. Confident AI prices on ingest volume at $1/GB-month, per its own comparison page (https://www.confident-ai.com/knowledge-base/compare/10-llm-observability-tools-to-evaluate-and-monitor-ai-2026). Under that model, verbose span payloads are a line on your invoice.

---

4. Tool and function-call telemetry, including arguments and outcome

The defining span type of agentic systems. The current #1 result never mentions it.

A tool span records the tool name, full arguments as structured JSON, the raw result, whether the call errored, the retry count, and the duration.

What this catches is tool selection quality. The model picked a plausible but wrong tool, or the right tool with a malformed argument. Both produce a valid-looking call and a wrong outcome. Galileo has productised this framing (https://galileo.ai/blog/best-llm-observability-tools-compared-for-2024). Borrow the concept. It is a lens, not a benchmark. Scoring the whole action sequence rather than each call is trajectory evaluation, covered in agent evaluation metrics.

The MCP wrinkle. When tools are served over Model Context Protocol, the tool boundary is also a process boundary. Span linkage requires propagating trace headers into the MCP call. Skip it and everything downstream of the invocation disappears from your tree. The same applies to sub-agent handoffs in LangGraph, CrewAI, AutoGen, LlamaIndex and the OpenAI Agents SDK: the child needs the parent span ID passed in explicitly, or it starts a fresh root.

Payload and secrets. Tool arguments are the most common place API keys, bearer tokens and customer PII leak into a trace store. They are user-influenced, structurally opaque, and rarely reviewed.

---

5. Retrieval provenance: what was fetched, ranked and actually put in the prompt

Every comparison table in this category has a “RAG support ✓” cell. None say what to capture.

Record the rewritten query, not just the user’s original. Record top-k document IDs, similarity scores, reranker scores and chunk boundaries, whatever the store is: Pinecone, Weaviate, Qdrant, Chroma or pgvector. Then record the one everybody skips, the final assembled context string that reached the model after truncation.

The most common silent RAG failure is that the right chunk was retrieved at rank 7 and truncated out during assembly. Recall@10 is fine. The model never saw it. Without an assembled_context attribute you will spend a sprint blaming the model for a string-concatenation bug. Debugging RAG retrieval failures walks that trace step by step.

Three retrieval measures are worth scoring, each on a different span:

Measure	What it asks	Attach to
Context relevance	Did retrieval fetch documents that could answer this?	The retrieval span
Groundedness / faithfulness	Is the answer supported by what was retrieved?	The generation span
Answer relevance	Does the answer address what was asked?	The terminal span

This triad was popularised by TruLens (https://openobserve.ai/blog/llm-observability-tools/). Ragas implements the same three measures as an offline library, which is why it keeps appearing on tool lists it doesn’t belong on.

Over a longer horizon, watch embedding drift: query and document distributions shifting as your corpus and users change. Record the embedding model too, since a swap from OpenAI to Cohere or Voyage AI invalidates every stored vector. Arize Phoenix inherits drift detection from Arize’s ML monitoring heritage, which predates LLMs entirely.

---

6. Prompt, model and code version stamped on every span

Record the prompt template ID and version. Record the model ID including the dated snapshot (claude-haiku-4-5-20251001, not claude-haiku). Record temperature and sampling parameters, the retriever index version, and your git SHA.

The payoff is a change in the shape of incident statements. “Quality dropped Tuesday” is not actionable. “Quality dropped on prompt v14, for enterprise-tier tenants, on the snapshot deployed at 14:02” is a fix.

There is a live hazard for reasoning models. Langfuse issue #11109 reports that Experiments and Evaluators fail to extract thinking and text blocks from Anthropic models with thinking enabled, with 62 reactions. If your capture layer can’t parse the response shape, your version-to-quality mapping is broken for exactly the models you are most likely to be evaluating.

Pin dated snapshots in production and treat a snapshot change as a deploy requiring an eval run. Anthropic, OpenAI, Google Vertex AI, Azure OpenAI and AWS Bedrock all publish dated snapshots because aliases move. An alias change with no corresponding commit in your repo is the hardest regression class to diagnose, because nothing in your system changed.

---

7. Session and multi-turn grouping

A session or thread ID joins N traces into one conversation. The quality questions here differ from trace-level questions: context retention across turns, contradiction with an earlier answer, tone drift, whether the user had to repeat themselves.

Two operational metrics belong here and appear in no competing page:

• Turns-to-resolution. Use the distribution. A bimodal shape (resolved in 2 turns, or never) tells you something an average hides.
• User rephrase rate. The share of turns where the user restated the same intent in different words. A strong implicit failure signal, available without a feedback widget.

Scoring at this level requires a tool that can evaluate a group, not a trace. Braintrust shipped Group scope for online scoring, keyed on a session key, with scores written either to the first trace or to every trace with prior turns as context (https://www.braintrust.dev/changelog). Ask vendors about that capability by name.

Confident AI’s own comparison page concedes that session grouping, which Langfuse and LangSmith both offer, is not cross-turn evaluation. Joining traces under a thread ID is table stakes. Scoring the thread as a unit is not.

---

8. Quality scores attached at the right granularity

The rule, which nobody else states plainly:

Score	Attach to
Context relevance	The retrieval span
Faithfulness to retrieved context	The generation span
Task completion	The terminal span of the trace
Coherence, resolution	The session / thread

Score everything at trace level and retrieval failures become invisible. A low overall score tells you the answer was bad. It does not tell you the retriever fetched documents about the wrong product.

Online versus offline. Online evaluation runs on sampled live traffic, cheap and continuous. Offline evaluation runs the full suite against a golden dataset in CI. They should share scorer definitions. If they don’t, your CI gate measures something different from your production alert, and you will ship a change that passes CI and pages you at 3am.

There is a sampling trap that costs teams weeks. Comparing two judges on two different random samples produces noise, not a result. Langfuse shipped consistent evaluator sampling, which compares evaluators on the same sample of matching observations (https://langfuse.com/changelog).

Golden datasets deserve more care than they get. Build them from production traces, not from imagination. Author expected outputs deliberately, with a named owner per case. Watch for the failure mode where the dataset ossifies around bugs you already fixed, so the suite penalises correct behaviour and nobody notices, because the number went down rather than up. Building golden datasets covers the selection procedure.

Judge scores need a reliability check before anyone alerts on them: agreement with human annotation on a held-out slice. A judge that agrees with your annotators 62% of the time is a random number generator with a confidence interval. Validating LLM judges covers the kappa calculation and the slice size you need.

---

9. Errors, retries, fallbacks and termination reason

Zero of the ten ranking pages tell you to record a terminal state. Without one, “success rate” is not a measurable quantity.

Record it on every trace as an enum:

• completed
• max_steps_exceeded
• tool_error_abort
• guardrail_blocked
• user_abandoned
• timeout

The runaway loop. An agent that re-plans indefinitely burns tokens without ever erroring. No individual step trips a timeout, and every span returns cleanly. The detection signals are step count per trace and cost per trace, treated as distributions. Alert on the p99. The mean stays stable while a small tail eats your budget.

Fallbacks as first-class events. Record provider failover, model downgrade and cache hit as explicit span events. A silent downgrade to a cheaper model during a provider incident is a quality incident that shows up on the finance dashboard as a cost win. Somebody will congratulate you for it.

Gateway-style tools (Helicone, Portkey, LiteLLM) see retries and failover natively, because they sit on the HTTP path. Galileo’s comparison fairly notes the corresponding limit: a proxy intercepting at the HTTP boundary cannot see agent reasoning or in-process sub-calls (https://galileo.ai/blog/best-llm-observability-tools-compared-for-2024).

---

10. User feedback and business metadata linked to the trace

Metadata makes traces sliceable. Attach it at ingest: user ID, tenant or org ID, environment, release, feature flag variant, plan tier, locale, channel. Backfilling onto historical traces is somewhere between painful and impossible.

Four feedback types, which the Langfuse post catalogues correctly: explicit ratings and comments, implicit behavioural signals, human annotation in a review queue, and automated judge scores from signal 8.

The join mechanic that post skips: feedback arrives asynchronously, sometimes hours after the trace closed. Your tool must support attaching a score to an existing trace by ID, after the fact. If it can’t, your only option is buffering feedback in your own database and giving up on slicing traces by it.

Four implicit signals are worth instrumenting, and none appear in a competing page. Copy events are a strong positive. Regeneration clicks are a strong negative. Edit-after-accept means the answer was nearly right, and the diff tells you how it was wrong. Abandonment mid-stream is usually a latency verdict, or a relevance verdict rendered in the first two sentences.

PostHog is the one tool in the field that natively joins LLM traces to product analytics and session replay (https://openobserve.ai/blog/llm-observability-tools/).

---

A worked example: what a fully instrumented 10-step agent trace looks like

One trace, one request: “Where’s my refund for order 44812?” Attributes aligned to OpenTelemetry GenAI semantic conventions are marked otel; custom attributes are marked custom. Verify attribute names against the live spec before adopting. They have changed more than once. Our OTel attribute reference tracks the deltas.

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "session_id": "sess_9f2c",              // custom: joins this to 5 other traces
  "user_id": "u_88213",                    // custom: signal 10
  "tenant_id": "acme-corp",                // custom
  "release": "api@7f3c9d1",                // custom: signal 6, git SHA
  "environment": "production",             // custom
  "termination_reason": "completed",       // custom: signal 9
  "spans": [
    {
      "span_id": "01", "parent": null, "name": "guardrail.input",
      "duration_ms": 38,
      "guardrail.policy_version": "v3",           // custom
      "guardrail.verdict": "pass",                // custom
      "guardrail.checks": ["pii", "prompt_injection"]  // custom
    },
    {
      "span_id": "02", "parent": "01", "name": "intent.classify",
      "gen_ai.system": "anthropic",                       // otel
      "gen_ai.request.model": "claude-haiku-4-5-20251001",// otel: dated snapshot
      "gen_ai.request.temperature": 0,                    // otel
      "gen_ai.usage.input_tokens": 412,                   // otel
      "gen_ai.usage.output_tokens": 6,                    // otel
      "unit_price_input_per_mtok": 1.00,                  // custom: signal 3
      "unit_price_output_per_mtok": 5.00,                 // custom
      "prompt.template_id": "intent_v9",                  // custom: signal 6
      "duration_ms": 310, "ttft_ms": 240                  // custom: signal 2
    },
    {
      "span_id": "03", "parent": "01", "name": "query.rewrite",
      "gen_ai.request.model": "claude-haiku-4-5-20251001",
      "query.original": "Where's my refund for order 44812?",  // custom
      "query.rewritten": "refund status order 44812 policy timeline", // custom: signal 5
      "duration_ms": 402
    },
    {
      "span_id": "04", "parent": "03", "name": "retrieval.embed",
      "embedding.model": "text-embedding-3-large",  // custom
      "embedding.cache_hit": false,                 // custom
      "duration_ms": 121                            // signal 2: embed time isolated
    },
    {
      "span_id": "05", "parent": "03", "name": "retrieval.vector_search",
      "retrieval.index_version": "kb-2026-07-30",       // custom: signal 6
      "retrieval.top_k": 20,                            // custom
      "retrieved_chunk_ids": ["kb_9921","kb_4410","kb_7732","…"], // custom: signal 5
      "retrieval.scores": [0.83, 0.81, 0.79],           // custom
      "duration_ms": 64
    },
    {
      "span_id": "06", "parent": "03", "name": "retrieval.rerank",
      "rerank.model": "rerank-v3",                  // custom
      "rerank.kept": 4, "rerank.scores": [0.94, 0.71, 0.58, 0.55],
      "assembled_context_chars": 6104,              // custom: signal 5, THE one people skip
      "assembled_context_truncated": true,          // custom
      "duration_ms": 188
    },
    {
      "span_id": "07", "parent": "01", "name": "tool.lookup_order",
      "tool.name": "lookup_order",                                  // custom: signal 4
      "tool.arguments": {"order_id": "44812", "include_refunds": true},
      "tool.result_status": "ok",
      "tool.retry_count": 0,
      "tool.transport": "mcp",                                      // custom
      "duration_ms": 233
    },
    {
      "span_id": "08", "parent": "01", "name": "generation.answer",
      "gen_ai.system": "anthropic",
      "gen_ai.request.model": "claude-sonnet-4-5-20250929",
      "gen_ai.usage.input_tokens": 7412,
      "gen_ai.usage.output_tokens": 218,
      "gen_ai.usage.cached_input_tokens": 5200,     // custom until standardised
      "prompt.template_id": "support_answer_v14",   // custom: signal 6
      "duration_ms": 2140, "ttft_ms": 610
    },
    {
      "span_id": "09", "parent": "01", "name": "guardrail.output",
      "guardrail.verdict": "pass",
      "guardrail.checks": ["pii_egress", "policy_claims"],
      "duration_ms": 44
    },
    {
      "span_id": "10", "parent": "01", "name": "feedback.attach",
      "feedback.type": "implicit",                  // custom: signal 10
      "feedback.event": "regenerate_clicked",
      "feedback.received_offset_ms": 41200          // arrived 41s after trace close
    }
  ],
  "scores": [
    {"span_id": "06", "name": "context_relevance", "value": 0.42},  // signal 8
    {"span_id": "08", "name": "faithfulness",      "value": 0.91},
    {"trace":  true,  "name": "task_completion",   "value": 0}
  ]
}

Read the scores. Faithfulness is 0.91, so the model was scrupulously faithful to what it was given. Context relevance is 0.42, so what it was given was mostly wrong. Task completion is 0. Trace-level-only scoring would report “bad answer” and send you to the prompt. Span-level scoring sends you to the reranker, which kept four chunks and truncated the assembled context at 6,104 characters.

Now break it. Insert an async context loss between spans 03 and 04. The same request renders as three trees: 01 to 03 under the original trace ID, 04 to 06 as a rootless fragment, 07 to 10 as a third. Trace count triples. Average spans-per-trace drops from 10 to 3.3. Cost-per-trace looks like it fell by two-thirds. Every dashboard improves, and you can no longer answer a single question about a single request.

That is why signal 1 comes first.

---

Where instrumentation actually breaks: a failure catalogue from public issue trackers

Frame this honestly. Open issues on a fast-moving open-source project are a sign of adoption, not of a bad product. Langfuse has by far the most evidence below because it has by far the most users and develops in public. A closed-source competitor with identical bugs would show you nothing. The value is knowing which edges to test in a proof of concept before you commit budget.

Issue	Project	What it reports	Reactions	Signal threatened
#9618	Langfuse	Incompatibility with Python 3.14, traced to Pydantic v1 usage	179	All: you can’t instrument what won’t import
#8780	Langfuse	”Failed to detach context” with LangChain CallbackHandler on async methods (ainvoke); affects LangGraph	62	1: context propagation
#11109	Langfuse	Experiments and Evaluators fail to parse Anthropic thinking blocks	62	6 and 8: version lineage, scoring
#5704	Langfuse	Dynamic imports break Jest tests without --experimental-vm-modules	57	Dev ergonomics; blocks the CI canary
#2169	Langfuse	Python SDK fully typed but ships no py.typed marker, so PEP 561 mypy checking fails downstream	51	Instrumentation reliability
#11874	Langfuse	Filtering traces by numeric score values returns wrong or incomplete results	50	8: score-based triage
#3961	Langfuse	FastAPI StreamingResponse resets ContextVars after the first chunk; one request becomes several traces	49	1: context propagation
#5653	Helicone	Self-hosted Playground returns “Invalid session” on /v1/playground/generate despite an authenticated dashboard	47	Self-host viability
#6572	Langfuse	ClickHouse CPU spikes after the V3 upgrade even with web and worker containers shut down and no platform usage	42	Self-host operating cost
#4555	Langfuse	Images uploaded via LangfuseMedia reach S3 but render as a button rather than inline	41	Multimodal debugging

Visual needed: Failure-mode catalogue table as a designed comparison graphic, showing issue title, project, number and URL, reaction count, open/closed status, date checked, and the signal it threatens. Data: the issue set above, each re-opened on the publication date to confirm status and whether a fixing release has shipped.

The Helicone self-hosting cluster

One failed self-host issue is a bad afternoon. Eight, spanning years, is a finding about the shape of the product.

• #1222: self-deploy docker compose fails following the docs
• #2284: compose env and MinIO errors
• #3080: numerous errors on a clean local compose
• #4332: supervisord failure in helicone-all-in-one
• #5646: jawn build error
• #5653: Playground “Invalid session” (47 reactions)
• #5657: the documented docker run command fails out of the box
• #5658: a self-hosted deployment routes an internal link out to the hosted site

Eight independent reports spanning issue numbers 1222 to 5658. That range is the point: long-running, not a recent regression. Helicone’s hosted product is a different proposition and these issues say nothing about it.

Release cadence is its own signal. Helicone’s latest tagged release is v2025.08.21-1, dated 21 August 2025, against a last push twelve months later (https://github.com/Helicone/helicone). Development is active. Roughly a year without a tag matters if your deployment policy pins to releases rather than tracking main.

The PoC checklist this catalogue produces

Before you sign anything, stand up the candidate and test six things against your stack:

1. Your actual async framework, with the concurrency pattern you really use.
2. Your streaming path, end to end, asserting one trace per request.
3. Your reasoning-model response shape, including thinking blocks.
4. Your serverless exit path. Kill the process mid-flush and count the spans that arrived.
5. Your score-filter workflow. Filter traces by a numeric judge score and verify the result set is complete.
6. If self-hosting: bring the stack up from the vendor’s published compose file, unmodified, and time it.

---

What it actually costs: a model from published pricing

Every listicle prints prices. None models them against a multi-step workload, which is where pricing structures diverge violently.

Scenario, held constant: 100,000 user requests per month, 10 spans per request (1,000,000 spans), average 6KB of span payload (roughly 6GB ingested), one seat where pricing is per-seat. Enterprise discounts and free tiers are not modelled.

Vendor	Published pricing model	Rough monthly bill at 1M spans / 6GB	Structural behaviour as steps grow
Langfuse Cloud	$29/mo for 100k events, then $8 per additional 100k	~$101	Scales linearly with steps, not requests
Comet Opik	$39/mo for 100k spans, then $5 per additional 100k	~$84	Linear in steps
Confident AI	$1/GB-month; Starter $200/mo incl. 5 GB-months	~$201	Scales with payload size, not step count
Datadog LLM Obs	From $8 per 10K LLM requests/mo (annual), 100K minimum	~$80 at 100k requests	Priced per request, so step count is free
LangSmith	Plus from $39/seat/mo plus volume	Seat floor plus usage	Seats flat; volume component still step-sensitive
Helicone	Pro $79/mo; Team $799/mo	$79+	Tier-based
Arize AX	From $50/mo	$50+	Tier-based
W&B Weave	Seat-based, roughly $39 to $50/seat/mo plus volume	Seat floor plus usage	Seats flat

Prices are derived from the Confident AI comparison page and the OpenObserve guide as read on the verification date. Re-verify every figure on the vendor’s own pricing page before budgeting. [VERIFY]

The structural point nobody makes

Going from a 3-step chain to a 10-step agent triples an event-priced bill and does not move a seat-priced or request-priced one at all. On the Langfuse Cloud rate card, 300k events costs about $45 a month and 1M costs about $101. Add a reflection loop that re-plans twice on average and you double that again. Add a judge running on 100% of traffic and every judge call is another billable span.

This is the largest cost variable in the category, and it is invisible if you compare vendors per request.

Visual needed: Cost curve chart, monthly bill (y) against steps-per-request (x: 1, 3, 5, 10, 20) at a fixed 100k requests/month, one line per pricing model (event-priced, GB-priced, seat-priced, request-priced). Footer must state the scenario assumptions and the date prices were captured.

Self-hosting, honestly

Langfuse self-hosted requires Postgres, ClickHouse, Redis and S3-compatible object storage, per Galileo’s comparison. The real comparison is not vendor bill versus zero. It is vendor bill versus a managed ClickHouse bill, plus object storage, plus Kubernetes and Terraform time to run four stateful services. We break the line items down in self-hosting Langfuse costs.

Langfuse issue #6572 reports ClickHouse CPU spiking after the V3 upgrade with containers shut down and no platform usage, with 42 reactions. Direct evidence the ClickHouse line item is not trivial even at idle.

The two levers that actually cut the bill

1. Payload truncation on the largest attributes. Assembled context strings and tool arguments dominate GB-priced bills. Truncate to a fixed prefix with a stored hash, and capture the full payload only on errored or low-scoring traces.
2. Tail-based sampling of successful traces with a 100% floor on errors, guardrail blocks and low scores. Sampling 10% of clean traffic typically cuts volume by an order of magnitude and improves the debugging dataset, because the retained set is enriched for what you actually investigate.

---

The 10 tools, mapped to the 10 signals

Name the conflict of interest first. Four ranking pages are vendor listicles that rank themselves #1. Their factual sections are often useful and are cited throughout. Their rankings are not evidence.

Verified repository facts:

• langfuse/langfuse: 32,717 stars, 756 open issues, licence field reported as NOASSERTION, latest release v4.6.0. https://github.com/langfuse/langfuse
• confident-ai/deepeval: 17,471 stars, 427 open issues, Apache-2.0, latest release v4.1.5. https://github.com/confident-ai/deepeval
• Helicone/helicone: 6,045 stars, 158 open issues, Apache-2.0, latest tagged release v2025.08.21-1. https://github.com/Helicone/helicone

Corrections to the field, offered so they can be checked rather than believed. Confident AI’s page states Langfuse has “21,000+ GitHub stars”. The repository shows 32,717. OpenObserve repeats “21,000+”. Both understate by roughly a third. The two pages contradict each other on Helicone’s licence: Confident AI says Apache-2.0, OpenObserve says MIT, the repository says Apache-2.0. “Langfuse is MIT” is repeated everywhere, while GitHub reports the licence field as NOASSERTION, consistent with an MIT core plus separately-licensed ee/ directories. If you have a legal review to pass, spell that out before you get there.

The ten

1. Langfuse. SDK plus OTel ingestion, capturing signals 1 to 8 and 10 natively. Session grouping is present, cross-turn evaluation less so. Self-host needs Postgres, ClickHouse, Redis and S3; cloud is free at 50k events/mo, then $29/mo. Best for a team that wants the full trace-plus-eval surface without lock-in. Now owned by ClickHouse; roadmap inferences from that are speculation.

2. LangSmith. SDK, deepest inside LangChain and LangGraph, strong on signals 1, 2, 6, 7 and 8 when your orchestration is LangChain. Proprietary, Plus from $39/seat/mo, and self-hosting is Enterprise-only, which Confident AI’s own table concedes. Best for teams committed to LangGraph.

3. Arize (Phoenix and AX). OpenInference instrumentation, OTel-compatible, with real drift monitoring inherited from Arize’s ML lineage, which makes it strongest on signal 5’s long-horizon drift. Phoenix is Elastic License 2.0: source-available, not OSI open source. AX from $50/mo.

4. Braintrust. SDK with Brainstore as a purpose-built backend. Group scope online scoring keyed on a session key is a genuine signal-7 capability few competitors match. Proprietary. Best for eval-heavy workflows where multi-turn scoring is the requirement.

5. Confident AI and DeepEval. DeepEval is the Apache-2.0 evaluation framework at 17,471 stars; Confident AI is the hosted platform. Strongest on signal 8, with a large metric library and pytest-style authoring. $1/GB-month, Starter $200/mo. Best for evaluation-first teams that will control payload size.

6. Galileo. Proprietary, with Luna-2 small language models as evaluators and an Agent Control layer. Productises tool selection quality, mapping directly to signal 4. Best for teams that want managed evaluators rather than running judge models. Its published numbers are audited below.

7. Helicone. Proxy-first, instrumented by changing a base URL. Sees signals 2, 3 and 9 natively at the HTTP boundary, including retries and failover, and cannot see in-process reasoning or sub-call structure. Apache-2.0, Pro $79/mo, Team $799/mo. Self-host has the multi-year bring-up cluster above.

8. Comet Opik. SDK, Apache-2.0, broad framework integration. Free at 25k spans/mo, then $39/mo for 100k plus $5 per additional 100k, making it one of the cheaper event-priced options at agent-scale span counts.

9. W&B Weave. SDK, Apache-2.0, inside the Weights & Biases ecosystem. Roughly $39 to $50/seat/mo plus volume, so step count doesn’t drive the seat component. Best for teams already running W&B for training.

10. OpenTelemetry with OpenLLMetry or OpenInference. Vendor-neutral, Apache-2.0. Instrument once against GenAI semantic conventions, export OTLP anywhere, swap backends without touching application code. Traces land in Jaeger, Grafana Tempo, SigNoz, Honeycomb, Datadog, New Relic, Dynatrace, Splunk or Elastic alongside the rest of your telemetry, with metrics in Prometheus and errors in Sentry. You build the evaluation layer yourself.

Adjacent picks worth knowing: LangWatch, Langtrace, Traceloop (which maintains OpenLLMetry), OpenLIT, HoneyHive, Literal AI, Lunary (free to 10k events/mo), PostHog (free to 100k LLM events/mo, 30-day retention), AgentOps, TruLens, MLflow, DSPy for prompt programs that need their own versioning, Portkey and LiteLLM as gateways, and OpenObserve, which is AGPL-3.0. OpenAI Evals and Ragas appear on at least one ranking listicle as “observability tools”. Neither does production tracing.

Visual needed: Signal-coverage matrix, 10 tools as rows and the 10 signals as columns, each cell marked native / partial / requires custom work / not available, with a footnote linking the specific documentation page establishing each rating. Ratings must come from reading official docs directly, not from vendor comparison tables.

The decision that actually matters: proxy vs SDK vs OTel

	Proxy (Helicone, Portkey)	SDK (Langfuse, Braintrust, LangSmith)	OTel plus OpenLLMetry / OpenInference
Setup	Change a base URL	Add instrumentation to app code	Add instrumentation, configure a collector
Sees in-process reasoning	No	Yes	Yes
Sees sub-agent structure	No	Yes	Yes
Sees retries and failover	Yes, natively	Only if you record them	Only if you record them
Backend portability	Vendor-coupled	Vendor-coupled	Portable
Evaluation layer	Limited	Included	You build it
Time to first value	Minutes	Hours to days	Days

For a multi-step system, choose OTel or an SDK. A proxy alone cannot reconstruct the span tree, because the tree exists inside your process and the proxy only sees what leaves it. Use a proxy as a complement for cost control and failover. Gateways vs observability platforms goes deeper on the boundary.

Visual needed: Instrumentation decision tree, branching on whether you need in-process reasoning visibility, whether you can change application code, whether you must self-host, and whether you need backend portability. Data: instrumentation model for each tool from its own docs, plus the stated architectural limits of proxy interception.

---

Vendor claim audit: which numbers here are actually verifiable

State the claim, name the vendor, name what verification would require, deliver a verdict. Never restate a vendor number in your own voice.

Claim as published	Vendor	Source	What verification would require	Verdict
”80x faster query performance compared to traditional databases” (Brainstore)	Braintrust	braintrust.dev	Named baseline database, query workload, dataset size, hardware	Unverifiable as published
”Teams report 30% accuracy improvements within weeks”	Braintrust	Same	Sample size, accuracy metric, baseline, selection method	Unverifiable as published
”Development velocity increases up to 10x”	Braintrust	Same	Any definition of velocity and a measurement window	Unverifiable as published
Luna-2 SLMs: “152ms average eval latency”	Galileo	galileo.ai	Prompt length, hardware, batch size, percentile or mean	Unverifiable as published
”97% lower cost than LLM-based evaluation”	Galileo	Same	The comparator model, since the denominator is unstated	Unverifiable as published
”~140x lower storage costs in typical log workloads vs Elasticsearch-based stacks”	OpenObserve	openobserve.ai	Dataset, compression settings, Elasticsearch config	Unverifiable as published, but caveated
”Fewer than 10% of organizations have successfully scaled AI agents into any business function”	Galileo, citing McKinsey	Same	A link to the McKinsey publication, which is not provided	Cite the primary report directly or omit [NEEDS SOURCE]
“A CHI 2025 study with 30 developers” underpinning four design principles	OpenObserve	Same	Paper title, authors, DOI	Unverifiable as published [NEEDS SOURCE]

Braintrust frames its numbers as “not marketing claims”, which raises the bar for methodology rather than lowering it. OpenObserve deserves partial credit for caveating that results vary by data entropy and cardinality.

Where a vendor page characterises a rival (“the LLM evaluation layer is shallow”, “observability depth drops outside LangChain”), that is competitive positioning, not a finding.

On instrumentation overhead. Helicone claims sub-millisecond proxy overhead. Braintrust claims minimal SDK overhead. No independent measurement of added p50 or p99 latency, or of dropped-span rate under load, exists for proxy versus SDK versus OTel collection. If overhead is load-bearing for your decision, run your own workload at production concurrency, with and without instrumentation, and compare p50, p99 and span-arrival completeness.

Visual needed: Vendor claim audit table as a designed graphic, with claim as published, vendor, source URL, what verification would require, and the verified or unverifiable-as-published verdict. Data: direct quotes and URLs from the Braintrust, Galileo, OpenObserve and Confident AI pages, plus a search for the McKinsey report and the CHI 2025 paper to determine whether either can be cited properly.

---

Sampling, retention, redaction: the operational decisions nobody writes about

Sampling

Sample whole traces, never individual spans. Span-level sampling produces partial trees, which are worse than no data because they look complete.

	Head-based	Tail-based
Decision point	At trace start	After the trace completes
Cost	Cheap, no buffering	Requires buffering the full trace
Knows the outcome	No	Yes
Keeps all errors	Only by luck	By rule

Retention

Competing pages cite 14-day base retention (LangSmith, Arize AX), 400-day extended retention on LangSmith, 30-day retention (Lunary, PostHog), and 60-day on the Opik free tier. Verify each against current documentation before relying on it [VERIFY].

One rule cuts across all of them. Keep full payloads hot for your debugging window. Keep derived metrics and scores forever. Trend analysis needs years of scores. It does not need years of 6KB prompt bodies, and those bodies are what your retention bill is made of.

PII, redaction and security signals

The highest-risk fields, in order: tool arguments, retrieved chunks, raw prompts.

Redact at the SDK, before export. A server-side redaction rule means the data already crossed your trust boundary and landed in a third party’s storage. Deleting it afterwards is remediation, not prevention. OpenLLMetry ships privacy controls for redacting sensitive prompts, which is the right architectural position for the control.

Security belongs in the trace, not in application logs. OWASP’s LLM Top 10 puts prompt injection first, so record the injection classifier score, the guardrail policy version and the blocking verdict as span attributes. Injection attempts per thousand traces then becomes a chartable number instead of a grep.

Data residency and compliance

Four deployment shapes exist: full self-host, hybrid data-plane and control-plane, VPC deployment, and regional cloud. “Self-hosting available” frequently means “Enterprise tier only”, which Confident AI’s own table concedes for LangSmith. Kubernetes operators exist for Langfuse and Phoenix; neither removes the ClickHouse operating burden.

Auditors ask different questions than engineers do. GDPR and HIPAA ask where payloads live and for how long, which is a retention and residency answer. SOC 2 asks who can read them, which is an access-control answer on your trace store. The EU AI Act’s transparency obligations ask you to show which model version produced a given output for a given user, which is signal 6 plus a retention policy. Build the version stamp now and the compliance answer is a query rather than a project.

Alerting hygiene

Alert on sustained score deltas across a statistically meaningful window. Never on a single trace. A judge score is noisy at n=1, and the channel gets muted within a week of firing on individual traces, at which point you have negative observability. Confident AI’s changelog records alerts gaining four priority levels so integrations can filter (https://confident-ai.com/docs/changelog). Route critical to PagerDuty, everything else to Slack, and review the warning tier weekly.

---

A 3-week rollout for a system already in production

Week 1, structure. Instrument signals 1 to 3 across the chain: context propagation, per-step latency, per-span cost with the unit price recorded at call time. Add the CI canary asserting span count and tree depth. Confirm flush-on-exit in every runtime you deploy to, including the ones you forgot about. Nothing else happens until the tree reconstructs reliably.

Week 2, semantics. Add tool, retrieval and version attributes. Establish truncation and redaction rules before volume becomes a bill and before PII becomes a disclosure. Set up session grouping with a stable thread ID.

Week 3, the loop. Attach quality scores at the correct granularity. Wire explicit and implicit feedback, including the async attach-by-trace-ID path. Define your terminal-state enum and emit it on every trace. Build the first golden dataset from 50 to 100 real production traces, deliberately including failures you already know about.

Exit criterion. Any engineer can take a customer complaint, find the exact trace inside two minutes, and see which of the ten steps went wrong without adding a log line.

The ordering mistake to avoid. Teams start with evaluation metrics, because scores feel like progress and traces feel like plumbing. Six weeks later the traces turn out to be orphaned, so the scores can’t be attributed to a step, a prompt version or a tenant. The dashboard is full and useless. Structure first.

---

Frequently Asked Questions

What is observability in multi-step LLM systems?

It is capturing every input, intermediate step and output of an AI application as one causally linked trace, then attaching quality, cost and latency measurements to each step so failures can be located and reproduced. Four objects carry it: spans, traces, sessions, and scores.

What are the pillars of LLM observability?

Four. Tracing is the causally linked span tree. Evaluation is quality scores attached at span, trace and session level. Cost and usage metering is tokens and spend per span. Version lineage is the prompt version, dated model snapshot and git SHA stamped on every span.

What is the most popular LLM observability platform?

Use a measure and a date. By GitHub stars among LLM-specific tools, Langfuse leads at 32,717, ahead of DeepEval at 17,471 and Helicone at 6,045. Stars measure developer mindshare, not production deployments. Every “best” list currently ranking for this query was written by a vendor that ranked itself first.

How is LLM observability different from traditional APM?

APM asks whether the system responded. Observability in multi-step LLM systems asks whether the response was right, and which step made it wrong. An agent can return HTTP 200 on all ten spans in 4 seconds while retrieving the wrong document and calling the wrong tool. The failure is a broken causal chain or a silent partial success, not an exception.

How do you trace a multi-step LLM agent end to end?

Create one trace per user request. Emit a child span per step with an explicit parent ID. Propagate context manually across async, streaming and sub-agent boundaries rather than relying on implicit context vars. Flush before process exit in serverless. Assert span count and tree depth in CI with a canary trace. The propagation failures are documented in Langfuse #3961 and #8780.

Which LLM observability tools are open source?

Licences, with the caveats competing pages skip:

Tool	Licence	Caveat
Langfuse	MIT core	GitHub reports the repo licence field as NOASSERTION; ee/ directories are separately licensed
Helicone	Apache-2.0	One ranking page lists MIT; the repo says Apache-2.0
Arize Phoenix	Elastic License 2.0	Source-available, not OSI open source
DeepEval	Apache-2.0	The framework, not the hosted Confident AI platform
Comet Opik	Apache-2.0	None
Lunary	Apache-2.0	None
W&B Weave	Apache-2.0	None
PostHog	MIT	None
OpenObserve	AGPL-3.0	Copyleft implications for hosted use
OpenLLMetry	Apache-2.0	None

How much does it cost for an agent that makes 10 calls per request?

At 100,000 requests/month with 10 spans each, published list prices range from a flat seat fee to a few hundred dollars. It depends entirely on whether the vendor prices per event, per GB, per request or per seat. Event-priced tools scale with step count. Request-priced and seat-priced ones do not.

Should I use a proxy, an SDK, or OpenTelemetry?

A proxy is the fastest path to cost and request visibility and needs only a base-URL change, but it intercepts at the HTTP boundary and cannot see in-process reasoning. An SDK sees the full tree at the cost of vendor coupling. OpenTelemetry decouples instrumentation from backend and leaves you to build evaluation. For observability in multi-step LLM systems: OTel or an SDK, with a proxy as a cost-control complement.

How do I know if my traces are broken?

Four checkable symptoms. Traces containing one span where you expect ten. Multiple traces sharing a single user request ID. Span counts that vary run-to-run for identical input. Missing trailing spans on serverless. The fix is a CI canary that asserts expected span count and tree depth.

Can these tools evaluate multi-turn conversations?

Distinguish two capabilities. Session grouping, joining traces under a thread ID, is widely supported, including by Langfuse and LangSmith. Cross-turn evaluation, scoring coherence, context retention and resolution across the thread, is supported by far fewer. Braintrust shipped Group scope online scoring keyed on a session key.

---

Related reading

• LLM evaluation metrics explained: faithfulness, groundedness, answer relevance and when each applies
• LLM-as-a-judge validation: how to check a judge against human annotation before you alert on its scores
• OpenTelemetry GenAI semantic conventions: a practical attribute reference for LLM spans
• Building golden datasets: a step-by-step selection guide
• RAG pipeline debugging: finding the retrieval failure behind a wrong answer
• Self-hosting Langfuse: the real footprint (Postgres, ClickHouse, Redis, S3) and what it costs to run
• AI gateways vs observability platforms: what each one can and cannot see
• Agent evaluation metrics: tool selection accuracy, action completion and trajectory scoring
• How to price LLM features: from cost-per-token to cost-per-completed-task
• Langfuse vs LangSmith vs Braintrust: an independent, evidence-based comparison

---

Method and corrections

This article is built from primary public documents: GitHub repositories and issue trackers, vendor changelogs, and published price lists. No product was installed, run or benchmarked. Where a question requires hands-on measurement, instrumentation overhead being the clearest case, the article says so and describes what a reader would need to measure.

Repository statistics, issue statuses and prices were checked on the verification date stated at the top and are re-verified quarterly. Langfuse shipped v4.6.0 two days before that check, which is the cadence this category moves at.

Correction log: no corrections issued yet. Any fact updated after publication will be recorded here with its original value, the corrected value and the date.

Observability in multi-step LLM systems is not a dashboard you buy. It is ten signals you emit, in the order given here, starting with a trace context that survives every async boundary in your stack. Get the tree right and the rest is configuration. Get it wrong and every number downstream is a well-formatted guess.