How to Trace a LlamaIndex RAG App in 2026 - Three Ways, With Setup

Tracing a single OpenAI call is easy. Tracing a LlamaIndex app is a different problem, because one query is not one call - it is a pipeline of retrieval, reranking, and generation. When a RAG answer comes back wrong, you need to see each stage on its own, and that is exactly what a good trace gives you.

This guide shows three ways to instrument a LlamaIndex pipeline, with working setup for each, plus how to wire in eval hooks so your traces carry quality scores. If you want the concept first, see what LLM tracing is.

Why a RAG trace is a tree, not a line

A plain LLM app produces one span per call. A LlamaIndex query produces a tree:

• A parent query span.
• A child retrieval span - embed the question, search the vector store, return chunks.
• Often a rerank span.
• A child synthesis / LLM span - generate the answer over the retrieved chunks.

The whole reason to trace RAG is to see those stages separately. A bad answer with a clean retrieval span points at the model. A bad answer with junk in the retrieval span points at the retriever. Without the tree, you are guessing. That is the same retriever-or-generator diagnosis behind RAG evaluation metrics.

The three approaches

Approach	Setup	Vendor-neutral?	Best tool here
Native callback handler	One registration at startup	No	Langfuse / Opik
OpenTelemetry via OpenInference	OTel instrumentor setup	Yes	Arize Phoenix
Manual spans	Wrap stages yourself	Depends	Any

Most teams want the first. The second is for OTel-standardized stacks. The third is a fallback for custom pipelines.

Approach 1 - the native callback handler (recommended)

LlamaIndex ships an instrumentation system, and observability tools hook into it with a global handler. Register it once at startup and every query is traced automatically - no changes to your query code.

Both Langfuse and Opik provide a LlamaIndex integration built on this pattern. The setup shape is three steps (illustrative - verify against current docs, this category ships breaking changes monthly):

pip install langfuse llama-index
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."

# Register the handler once, at app startup (illustrative)
from langfuse.llama_index import LlamaIndexInstrumentor

LlamaIndexInstrumentor().start()

# Your normal LlamaIndex code - unchanged
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

docs = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(docs)
response = index.as_query_engine().query("What is the refund window?")

That single registration is the whole integration. Every query() from then on fans out into the retrieval and generation spans described above, with inputs, outputs, token counts, cost, and latency captured per stage. Opik works the same way with its native LlamaIndex integration and its own SDK.

Why this wins for most stacks - near-zero code, and the tool observes rather than sits in your request hot path, so a tracing backend hiccup does not break your query. The same argument we make for the SDK-wrapper pattern in how to trace OpenAI API calls.

Approach 2 - OpenTelemetry via OpenInference (the standards-clean route)

If your organization has standardized on OpenTelemetry, instrument LlamaIndex through the OpenInference conventions and emit standard OTel spans. Arize Phoenix is built on OpenTelemetry and its own OpenInference semantic-convention project, and it is framework-agnostic - it works well beyond LangChain, which makes it a natural fit for LlamaIndex. It also runs locally in under a minute, so you get a trace UI on your laptop fast.

The setup shape (illustrative):

pip install arize-phoenix openinference-instrumentation-llama-index llama-index

# Illustrative - register the OpenInference instrumentor
import phoenix as px
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor

px.launch_app()  # local trace UI
LlamaIndexInstrumentor().instrument()

# LlamaIndex code below is unchanged and now emits OTel spans

The payoff is portability - your instrumentation is standard OTel, so you can repoint it at any OTLP backend later without touching app code. Both Langfuse and Opik also expose OTLP ingestion, so you are not locked to Phoenix. The cost is that OTel is more setup than a one-line handler, so reach for it when OTel is already a requirement. One license note - the Phoenix server is Elastic License 2.0, source-available rather than OSI open source, which only bites if you plan to resell it as a hosted service. For internal use it behaves like open source. More context in how to set up LLM tracing.

Approach 3 - manual spans (the fallback)

If you have a heavily customized pipeline that does not fit the callback handler, you can wrap each stage in a span yourself using the tool’s SDK - a parent span for the query, child spans for retrieval and generation. This is the most code and the most error-prone, so treat it as a fallback for the cases the automatic instrumentation cannot reach.

Wiring in eval hooks

Tracing tells you what happened. Evaluation tells you how good it was. The power move is to attach scores to the spans you just captured. Once retrieval and generation are separate spans, run RAG metrics against each and attach the scores back to the trace - faithfulness on the generation span to catch hallucination, and context precision and recall on the retrieval span.

Both Langfuse and Opik support scoring traces this way, with LLM-as-judge scorers or custom functions, and both can run online evaluation on live production traffic - Opik markets online evaluation as a core feature, and Langfuse lets you attach LLM-as-judge and custom scores to any trace. That closes the loop - the trace shows the pipeline, the scores grade it, and together they tell you which stage to fix. The full workflow is in how to evaluate RAG.

Which tool for LlamaIndex tracing?

• You want the cleanest setup, actively developed, free to start - the native handler on Langfuse (MIT, self-host free, $29/mo cloud) or Opik (Apache-2.0, cheapest cloud at $19/mo, native LlamaIndex integration). This is the default.
• You have standardized on OpenTelemetry - instrument via OpenInference into Arize Phoenix or any OTLP backend, mindful of the ELv2 license on the server.
• You want tracing plus built-in online RAG scoring - Opik or Langfuse, which attach eval scores to the same spans.

Bottom line - trace LlamaIndex with a native callback handler, get the full retrieval-plus-generation span tree for one line of setup, then attach RAG scores so every trace carries a quality signal. For the wider field, see best LLM tracing tools and best RAG evaluation tools. Code shapes here follow each tool’s standard integration pattern - verify against current docs, and every price was read from each vendor’s own pages on 23 July 2026.