How to Set Up LLM Tracing in 2026 - A Practical Guide

Tracing is the first thing I add to any LLM app that has real users. Not evals, not dashboards - tracing. Because the moment a user reports a bad answer, the only question that matters is “what actually happened,” and without a trace you are guessing.

This is a practical guide to setting it up. I will cover what a trace captures, how to pick a backend, and the concrete steps to get your first trace flowing - with the three open-source platforms I reach for most.

What a trace actually captures

A trace is the full record of one request through your app, stored as a tree of spans. Each span is one step - an LLM call, a retrieval query, a tool invocation, an agent decision - and it carries the inputs, the outputs, the latency, the token count and the cost.

The point is the tree, not the log line. A flat log tells you an LLM call happened. A trace tells you that the user’s question triggered a retrieval that pulled three bad chunks, which fed a prompt that produced a confident wrong answer, in 4.2 seconds, for $0.03. That is the difference between staring at logs and actually seeing the failure.

For agents and RAG pipelines this is not optional. A multi-step agent can make a dozen model and tool calls per request. When one goes wrong, you need the tree to find it.

Step 1: Pick a backend

You have three good open-source options, and the right one depends on how much ops work you want to do.

• Arize Phoenix if you want to start in the next 60 seconds. It runs locally on a laptop in under a minute with a working trace UI, built natively on OpenTelemetry. No signup.
• Langfuse if you want the open-source default with a clean managed option. MIT-licensed, self-hosts free, and the Core cloud tier is $29/mo.
• Opik if you want the cheapest managed cloud with a permissive license. Apache-2.0, full feature set self-hosted, and Pro cloud is $19/mo for 100k spans.

All three are framework-agnostic and support the OpenAI SDK, LangChain and LlamaIndex, so the instrumentation looks similar whichever you choose. Start on the free tier or a local instance - you do not need to commit to hosting before you have seen your own traces.

Step 2: Instrument your app

There are two ways to get spans into a backend, and you will usually use both.

1. Auto-instrument the SDK. Wrap your model client and the platform captures every call automatically. Langfuse, Opik and Phoenix all support the OpenAI SDK wrapper, so a standard OpenAI or Anthropic app needs almost no manual work - you swap the import or add a decorator and calls start showing up traced.
2. Decorate your own functions. For the glue code between model calls - retrieval, tool handlers, post-processing - add a span decorator so those steps appear in the tree too. This is what turns a flat list of LLM calls into a real execution trace.

If you are on a framework, lean on the native integration. Langfuse and Opik both ship LangChain and LlamaIndex integrations that trace chains and agents with a single callback or handler. Opik advertises 60+ framework integrations for exactly this reason.

The common mistake here is tracing only the LLM calls. If you skip the retrieval and tool spans, you get a trace that shows the model was fed garbage but never tells you where the garbage came from. Instrument the whole request path, not just the model.

Step 3: Add context that makes traces searchable

A pile of traces is useless if you cannot find the one you need. Before you ship, attach metadata:

• A session or conversation ID so you can group a multi-turn chat into one thread.
• A user ID so a bug report maps to that user’s actual traces.
• Tags for the app version, the prompt version and the environment.

This costs you a few extra fields at instrumentation time and saves you hours later. When someone says “the bot gave me nonsense at 2pm,” you filter by their user ID and open the trace instead of scrolling.

Step 4: Verify it end to end

Before you call it done, run one real request and confirm the trace tree is complete. Check three things:

1. Every step appears. Retrieval, each model call, each tool call - all present and nested correctly.
2. Inputs and outputs are captured, not truncated or empty. If outputs are missing, your wrapper is not seeing the response.
3. Cost and tokens are populated. If they are zero, the model metadata is not being passed through, and your cost dashboards will lie to you.

One caveat worth knowing: Phoenix has reported ingest lag - traces can take a noticeable moment to appear in the dashboard. If your first trace does not show instantly, give it a few seconds before assuming the wiring is broken.

Step 5: Decide self-host or cloud

Once tracing works, decide where it lives long term.

Self-host if you have the ops capacity and data-residency needs. Langfuse is free under MIT and Opik is free under Apache-2.0 - both give you the real product on your own infrastructure. The honest catch on Langfuse is that its v3 self-host is four services (Postgres, ClickHouse, Redis and S3-compatible storage), so budget the setup time. Opik is designed to self-host at Kubernetes scale without losing features.

Use managed cloud if you would rather not run infrastructure. Opik Pro at $19/mo is the cheapest of the major platforms, and Langfuse Core at $29/mo is close behind. Both free tiers - Langfuse Hobby and Opik Free - are enough to trace a small production app before you pay.

The setup that actually pays off

Tracing is cheap to add and expensive to skip. The teams that regret it are the ones who waited until a production incident to wish they could see inside a request.

My default: start on Phoenix locally to feel how traces work, then move to Langfuse or Opik for a persistent backend once you are ready to keep history. Instrument the whole request path, tag everything, and verify the tree end to end before you trust it. Do that, and the next bad answer is a trace you open, not a mystery you debug blind.