How to Monitor an LLM in Production in 2026 - The Full Workflow

Most teams monitor their LLM app the way they monitor any web service - latency, error rate, uptime - and then get blindsided when quality quietly falls off a cliff. The model was fast the whole time. It was just wrong.

That is the thing about LLM monitoring: a bad answer returns HTTP 200 with normal latency. Classic application monitoring has no way to see it. A hallucinated response looks identical to a perfect one from the outside. So production LLM monitoring has two layers - the operational one you already know, and a quality layer that scores whether the output is actually good. Miss the second layer and you are flying blind on the only thing that matters. If you want the conceptual split spelled out, I covered it in LLM observability vs monitoring.

Here is the full workflow I run in production.

Layer 1: the operational metrics

Start with what you can measure cheaply on every request:

• Latency - end to end, and per step for multi-step apps.
• Error rate - failed calls, timeouts, provider outages.
• Token count and cost - per request, rolled up by feature.
• Throughput - requests per minute, so you spot spikes.

These are table stakes, and any good LLM observability tool captures them automatically once you are tracing. They tell you the system is healthy. They do not tell you it is correct.

Layer 2: output quality

This is the layer that separates LLM monitoring from APM. You score the actual outputs on live traffic - this is called online evaluation, and it is the core move. You sample real production requests and run metrics on them: hallucination, answer relevancy, toxicity, policy violations. Usually these run as LLM-as-judge, one model scoring another’s output.

Why online and not just offline test sets? Real user inputs are messier than anything you would write in a test suite, and quality drifts as prompts, models, and your data change underneath you. Online eval catches the regression that only appears in production.

The step-by-step workflow

1. Trace every request end to end. Instrument the app so each request logs the full path - prompt, retrieved context, tool calls, model response - as one connected trace. Multi-step agents especially need this, or you cannot tell which step broke. If you have not set this up yet, my LLM tracing setup guide walks through it.

2. Capture the operational metrics. Latency, cost, tokens, errors, attached to each trace automatically.

3. Turn on online evaluation. Sample a slice of live traffic - you do not need to score every request - and run quality metrics on it. Start with hallucination and relevancy, add safety checks as needed.

4. Add guardrails on the critical paths. For high-stakes outputs, score or filter before the response reaches the user, not just after. This blocks the worst outputs in real time rather than reporting them later.

5. Alert on regressions. Set thresholds on both layers - latency spikes and quality drops - so you hear about a problem before your users do.

6. Close the loop with human feedback. Route flagged or low-scoring outputs to a human annotation queue, then feed those labels back into your eval set. Your monitoring gets smarter over time instead of static.

7. Sample to control cost. Online eval fires extra model calls, so scoring 100% of traffic gets expensive. Sample a representative slice - the signal holds, the bill does not explode.

The tools I trust

Three platforms do this well, and the choice is mostly about hosting and license.

Langfuse is my default for most teams. It does tracing, cost tracking, online evals, prompt management, and human annotation in one system, it is framework-agnostic, and it maps the OpenTelemetry conventions. The big draw is that self-hosting is free under MIT with almost every feature intact, and it runs roughly 25x cheaper than LangSmith at high trace volume - so your monitoring layer is not itself a huge cost. The honest catch is operational: the v3 self-host needs Postgres plus ClickHouse, Redis, and S3-compatible storage - four services. If you cannot run that stack, the $29/mo managed Core tier sidesteps it.

Opik is the closest alternative and the cheapest managed option. It captures multi-step traces, runs online (production) evaluation with LLM-as-judge and code metrics, and ships Guardrails and an Agent Optimizer - so layers 1, 2, and 4 above are all native. The OSS build is Apache-2.0 with the full feature set self-hosted, and the Pro cloud is $19/mo for 100k spans, the cheapest paid cloud in the category. The watch-out is per-seat pricing on the cloud - it climbs as your team grows, so model the seat cost if you are a big team. Opik vs Langfuse compares the two head to head.

Arize Phoenix is the fastest to stand up. It is genuinely OpenTelemetry-native, so tracing is clean, and it runs locally in under a minute with 50+ pre-built eval metrics for the quality layer. Two things to know: the server is Elastic License 2.0, source-available rather than OSI open source, which only bites if you resell it as a service; and there are reports of ingest lag, so if real-time visibility is critical, test that on your traffic first.

So which one?

• You want the free open-source default and can run the self-host stack - Langfuse.
• You want managed monitoring for the least money, with online eval and guardrails - Opik at $19/mo.
• You want the fastest setup and a strict OpenTelemetry-native architecture - Arize Phoenix, as long as you are not reselling it.

Whichever you pick, the principle holds: monitor quality, not just health. For a ranked look at the wider field, see the best LLM monitoring tools roundup.