Observability and Tracing in Production

Everything in this course so far has happened before deployment - metrics, judges, pipelines, agent tests. Then you ship, and real users send inputs you never imagined. A prompt that passed every test produces a hallucination for one customer. Costs spike overnight. Latency doubles on Tuesdays. You cannot fix what you cannot see, and LLM systems are unusually hard to see into. Observability is how you get that visibility, and tracing is the mechanism underneath it.

Monitoring tells you what broke, observability tells you why

These two words get used interchangeably and they should not be. Monitoring is watching a fixed set of metrics you chose in advance: request rate, error rate, p95 latency, tokens per day, dollars per day. It answers questions you already knew to ask. When error rate jumps, monitoring pages you.

Observability is the broader ability to interrogate behavior you did not anticipate. Why did this specific request return nonsense? Which retrieval step pulled the wrong document? Was it a bad prompt version, a model change, or a user who pasted 40,000 tokens of garbage? You cannot pre-register every question, so instead you capture enough structured detail about every request to reconstruct and reason about any of them after the fact. Our overview of what LLM observability is unpacks this distinction in more depth, and the monitoring versus observability framing matters because teams that only monitor stay blind to the failures that hurt most.

Traces and spans, the core data model

The unit of observability is the trace: one complete request through your system. A trace is made of spans, which are nested, timed units of work. Picture a RAG chatbot answering a question. The trace for one message might look like:

• Trace: “user asks about refund policy”
  • Span: embed the query (12ms)
  • Span: vector search, returns 5 chunks (34ms)
  • Span: assemble prompt with retrieved context
  • Span: LLM call - model, input tokens, output tokens, cost, latency (1,840ms)
  • Span: post-process and format response

Each span records its inputs, outputs, duration, and metadata. This structure is what makes an opaque black box legible. When a response is wrong, you open its trace and see immediately whether retrieval returned junk, the prompt was malformed, or the model itself hallucinated over good context. Without tracing you would be staring at a single output string with no idea which stage failed. Our guide to LLM tracing walks through instrumenting these spans step by step.

Getting traces out of your app

There are two common paths to emit traces. The first is a vendor SDK: you wrap your LLM client or decorate your functions, and the SDK captures spans automatically. This is the fastest way to start, especially with framework integrations for LangChain, LlamaIndex, or the OpenAI SDK.

The second path is OpenTelemetry, the open standard for telemetry. You instrument once against OpenTelemetry, and any compatible backend can receive the traces. The payoff is portability - you are not welded to one vendor’s proprietary format. Increasingly the LLM ecosystem has converged on OpenTelemetry semantic conventions for generative AI, so this is the future-proof choice for a serious production system. The tradeoff is a bit more setup than dropping in a vendor SDK.

A useful middle ground many tools offer: a proxy. You point your model calls at their endpoint instead of the provider’s, and the proxy logs everything transparently with a one-line base-URL change. Helicone is built around exactly this proxy model, which is why it is often the fastest observability to stand up - no code changes to your call sites at all.

What to actually watch in production

Instrumentation is worthless if you never look at the right signals. The load-bearing ones:

Cost. Track tokens and dollars per request, per feature, and per user. LLM bills surprise teams constantly, and cost is often the first thing to drift when someone changes a prompt or model.

Latency. Measure it per span, not just end to end. A slow response might be slow retrieval, not a slow model, and the fix differs entirely.

Quality signals. Run the online evaluators from earlier chapters against a sample of live traffic. An LLM judge scoring production responses for relevance or hallucination turns observability from “here is what happened” into “here is what happened and how good it was.”

User feedback. Capture thumbs up and down, and link them to traces. A downvote attached to a full trace is one of the highest-signal debugging artifacts you can have.

Drift. Watch how input distributions and output quality change over time. A prompt that worked in July can degrade when a provider silently updates the underlying model.

Our LLM observability best practices and how to monitor LLM in production go deeper on setting thresholds and alerts around these signals.

How the real tools do it

Langfuse is the open-source anchor here - self-hostable, framework-agnostic tracing with nested spans, plus dataset experiments and online evaluators layered on top, so the same trace you debug can also be scored. Arize Phoenix comes from the ML observability world and is strong on OpenTelemetry-native tracing and drift analysis, with an open-source core you can run locally. Opik, from the Comet team, pairs tracing with an evaluation focus and is also open source. Helicone is the proxy-first option optimized for zero-friction setup and cost tracking. The right pick depends on whether you prioritize self-hosting, setup speed, or deep drift analytics, and that decision is exactly what the final chapter is about.

The pattern to internalize is that tracing and evaluation are not separate systems. The best setups make every production trace a candidate eval sample - you find a bad trace, add it to a dataset, and it becomes a regression test that protects you forever.

Key takeaways

• Monitoring watches metrics you chose in advance; observability lets you ask new questions about failures you never predicted.
• Traces made of nested spans turn an opaque LLM pipeline into something you can inspect stage by stage.
• Emit traces via a vendor SDK, a proxy, or OpenTelemetry - OpenTelemetry keeps you vendor-portable.
• Watch cost, per-span latency, live quality scores, user feedback, and drift, and feed bad traces back into your eval datasets.

Next up: Choosing Your Eval Stack, where we pull the whole course together into a decision you can actually make.