Observability

What it means

Observability for LLM applications means capturing enough signal about each request to reconstruct and understand what the system did. The core unit is the trace - a record of a single request broken into spans for each step, such as a retrieval call, a prompt rendering, the model call itself, and any tool invocations. Attached to each span you log inputs, outputs, token counts, cost, latency, and metadata like user or session.

Unlike traditional software, an LLM app cannot be stepped through in a debugger, and the same input can produce different outputs. Tracing is the substitute - it turns an opaque, non-deterministic pipeline into something you can inspect, search, and reason about after the fact.

Why it matters

You cannot improve what you cannot see. Observability is the foundation that everything else sits on. It is how you find the slow span driving a latency complaint, the prompt version that caused a quality regression, or the runaway loop burning tokens. It feeds evaluation by giving you real production traffic to build datasets from, and it makes guardrails auditable by logging every block. When investigating a prompt-injection incident, the trace is often the only record of what the model actually received and did.

In practice

Most platforms build on OpenTelemetry-style tracing so you are not locked into one vendor. Langfuse is the popular open-source default, giving you self-hostable traces, sessions, and cost tracking. Helicone takes a proxy-based approach that captures requests with minimal code changes, and LangSmith ties tracing tightly to evaluation for teams in the LangChain ecosystem. A sensible starting point is to instrument traces first, confirm you can find a single slow or failing request, then layer on evals and alerts. See LLM observability best practices for a fuller workflow and best LLM observability tools for a comparison.