What Is LLM Tracing? How to See Inside an AI Request in 2026

An LLM app used to be one function call: prompt in, answer out. Nobody needed tracing for that. But a 2026 AI app is a pipeline - embed the query, search a vector store, retrieve documents, call the model, run a tool, maybe loop through an agent a dozen times. When the final answer is wrong, which of those steps broke it? LLM tracing is how you find out: it records every step of a single request as a readable tree, so you can see exactly what happened instead of guessing.

The short version

A trace is the full record of one request. It is made of spans, and each span is one step. The spans nest - an agent span contains child spans for every tool it calls, a RAG span contains spans for the retrieval and the generation. Read together, they form a tree that shows the request in the order it actually executed, with the inputs, outputs, token counts and latency of each step attached.

Tracing is the foundation everything else in observability sits on. You cannot evaluate an output you did not capture, and you cannot debug a failure you cannot see. Traces are the raw material - evals score them, metrics summarize them, but tracing is what records reality in the first place. It is the difference between “the answer was wrong” and “the retriever pulled the wrong document on step three.”

If you want the wider frame, tracing is one of the three pillars in what is LLM observability - the other two being evals and metrics.

How it works

You add an SDK or an OpenTelemetry exporter to your app. Most tools auto-instrument the common providers and frameworks, so a single initialize call at the top of your program starts capturing calls to OpenAI, Anthropic, your vector store and your agent framework without you decorating every function. For anything custom, you wrap the function in a span decorator and it joins the tree.

From then on, each request produces a trace. The magic is not the recording - it is the tree structure. A flat log of “called model, called tool, called model” tells you little. A tree that shows the tool call nested inside the agent step that spawned it, with the exact input that tool received and the exact output it returned, tells you where the run went off the rails. You open a bad trace, read down the tree, and stop at the span whose output first looks wrong.

The OpenTelemetry connection

Here is the part worth understanding. OpenTelemetry (OTel) is the open standard for how traces are structured and shipped, and the good LLM tracing tools adopt it so you are not locked to one vendor. OTel came from the microservices world - it is how backend teams already trace distributed systems - and the LLM tools extend it with GenAI semantic conventions that describe model calls, tokens and tool use.

The practical payoff: if your tracing speaks OTel, you can point it at a different backend later without re-instrumenting your app, and your AI traces sit alongside the rest of your infrastructure traces. There is a depth difference, though. Some tools are OTel-native, built on it from the ground up. Others accept OTel as one ingestion path among many. Both work; the native ones lock you in less. We go deeper in OpenTelemetry for LLM observability.

Where the tools fit

Langfuse is the framework-agnostic default. It runs as an OpenTelemetry backend on an OTLP endpoint, maps the GenAI semantic conventions, and self-hosts free under MIT with almost every feature intact. It supports OTLP over HTTP, though not gRPC yet. If you want tracing plus evals plus prompt management in one open-source tool, it is the safe pick - just know the self-host stack is now four services (Postgres, ClickHouse, Redis and S3-compatible storage).

Opik is the cheapest managed way in. Comet’s open-source platform captures multi-step traces of LLM calls, tools and agent activity, is Apache-2.0 with the full feature set self-hosted, and its Pro cloud is 19 dollars a month for 100k spans. It has 60+ framework integrations plus an OTLP ingestion path - OTel is one path here, not the native architecture. Watch per-seat pricing as your team grows.

Laminar is built for agents, especially browser agents. It is OpenTelemetry-native, written in Rust for low overhead, and one line - Laminar.initialize() - auto-captures traces of LLM, tool and sub-agent calls. For browser agents it syncs session recordings with the agent steps on one timeline, and Browser Use documents it as their observability integration. It is the only platform of these you can self-host in full. The gotcha is billing: its “Signals” unit is metered by tokens spent reading your traces, which is hard to forecast - self-hosting removes that entirely.

For a ranked comparison see the best LLM tracing tools, and for the hands-on version, how to set up LLM tracing walks the actual wiring.

The bottom line

Tracing is the first thing to add and the thing everything else depends on. Wire in one-line auto-instrumentation, capture real requests as trees, and get in the habit of opening a bad trace instead of re-running the prompt and hoping. Pick an OpenTelemetry-friendly tool so you are not locked in, start with an open-source platform so it costs nothing, and add evals and metrics on top once the traces are flowing. See inside first; optimize second.