Online Evaluation

What it means

Online evaluation runs against real requests as they happen in production. Instead of a curated dataset, the inputs are whatever users actually send. Scoring can come from automated checks, an LLM-as-a-judge running on sampled traffic, user feedback signals like thumbs up and down, or human annotators reviewing a slice of live conversations. Because it observes real behavior, it catches issues that never appeared in your test data.

Why it matters

No offline test set fully represents production. Users phrase things in unexpected ways, distributions drift over time, and edge cases surface only at scale. Online evaluation is how you detect quality regressions, hallucinations and rising cost or latency after a release, and it feeds a continuous loop: real failures become new examples for your offline suite. Without it, you are flying blind between deploys.

In practice

Production-focused platforms are built around this loop. Langfuse and Braintrust let you attach scores and human annotations to live traces, run judge models on sampled requests, and alert when a metric slips. A common pattern is to score every request with cheap automated checks, sample a percentage for LLM-as-a-judge or human review, and promote interesting failures into a regression dataset. See how to monitor LLMs in production for concrete sampling and alerting strategies.