Offline vs Online Evaluation

Two questions, two modes

In the previous chapter we said every eval answers one of three questions. Those questions map onto two fundamentally different modes of evaluation, and confusing them is one of the most common mistakes teams make.

Offline evaluation happens before you ship. You run a fixed dataset through your system in a controlled setting and score the results. It is repeatable, deterministic in its setup, and gated by a version of your prompt or model that you control.

Online evaluation happens after you ship. You measure quality on live production traffic - real users, real inputs, no reference answers. It is continuous, noisy, and the only place you learn what actually happens in the wild.

You need both. Offline evals protect you from shipping known regressions. Online evals protect you from the failures you never imagined. A team with only offline evals has a confident test suite and no idea how the feature performs at 2am on a Tuesday. A team with only online evals sees problems but cannot safely fix them because every change is a live experiment. The observability vs monitoring post draws a related line worth reading alongside this.

Offline evaluation - the lab

Offline evaluation is your lab bench. The defining feature is a fixed dataset, so results are comparable across runs. Change nothing but the prompt, rerun, and any score movement is attributable to that change.

This is where regression testing lives. Just as you would never merge code without running the test suite, mature teams gate LLM changes behind an offline eval run in CI. Open a pull request that edits a prompt, the eval suite runs against your golden dataset, and the score delta shows up right in the PR. The how to run LLM regression tests guide walks through wiring this up.

Offline evaluation shines for:

• Comparing prompt versions. Does the new system prompt beat the old one on your 100 cases?
• Model migration. Run the same suite against two models, compare accuracy and cost.
• Catching regressions before merge. Block the PR if the score drops below a threshold.
• Reproducing bugs. A user hit a bad case? Add it to the dataset so it can never silently return.

The hard limit of offline evaluation is coverage. You can only test cases you put in the dataset, and no dataset covers the true diversity of production. That is precisely the gap online evaluation fills.

Online evaluation - the wild

Online evaluation measures quality on live traffic. There is no expected answer to compare against, so the methods are different. The three main techniques:

Reference-free scoring. You run a scorer - often an LLM-as-judge - on real outputs to rate qualities like helpfulness, groundedness, or toxicity without needing a ground-truth answer. Because you cannot score every request cheaply, teams usually sample a percentage of traffic.

Guardrails. A subset of online checks run inline and can block or rewrite a response before it reaches the user - for example, refusing to return output that contains PII or that fails a safety classifier. These must be fast and cheap because they sit in the request path.

User signals. Thumbs up and down, edits, retries, abandonment, and downstream task completion are all implicit quality labels. They are noisy but they reflect real user outcomes, and over time they become training data for better scorers - and new rows for your offline dataset.

Online evaluation depends entirely on having good production instrumentation. You cannot score traffic you did not capture. This is why online evaluation and observability are so tightly linked - you need tracing in place to have anything to evaluate. The how to monitor LLM in production post covers the plumbing.

The flywheel between them

The two modes are not separate silos - they feed each other, and the loop is where the real value compounds.

Production traffic (online) surfaces a failure mode you never anticipated. You capture that trace, curate it into a test case, and add it to your offline dataset. Your offline suite is now stronger. You fix the prompt, the offline suite confirms the fix and confirms nothing else regressed, and you ship with confidence. Then you watch the online metrics to verify the fix holds on real traffic.

Offline finds problems fast and cheap. Online finds problems you did not know existed. The handoff between them - production trace becomes test case - is the single most important habit in LLM evaluation. Teams that build this flywheel improve steadily. Teams that treat offline and online as unrelated projects stall.

How the real tools do it

Most serious platforms support both modes, but they lean in different directions:

• LangSmith spans the full loop - dataset-based offline evals plus tracing and online evaluators over production runs, with a UI to promote traces into datasets.
• Arize Phoenix comes from an observability heritage and is strong on online monitoring and drift detection, with evals layered on top of captured traces.
• Opik pairs a tracing backend with an evaluation library, so online traces and offline scorers live in one place - a natural home for the flywheel.
• Helicone sits as a proxy in front of your model calls, making it a low-friction way to capture the production traffic that online evaluation needs.

When you are choosing, ask which mode you are weakest in. If you ship blind, prioritize online capture first. If you break things on every change, prioritize offline gating. The best LLM eval tools for production roundup compares them on exactly these axes.

Key takeaways

• Offline evaluation runs a fixed dataset in a controlled setting - it is your lab for safe changes, model migration, and regression gating in CI.
• Online evaluation measures live traffic with reference-free scorers, inline guardrails, and user signals - it catches failures no dataset anticipated.
• Neither mode is optional - offline gives you safe iteration, online gives you real-world truth, and each covers the other’s blind spot.
• The flywheel is the point - production traces become offline test cases, and that handoff is the habit that separates teams that improve from teams that stall.

Next we go deep on the raw material both modes depend on in the building evaluation datasets chapter.