How to Evaluate Multi-Turn Conversations in LLM Apps (2026)

Your chatbot passes every single-turn test and still frustrates real users. That is not a paradox - it is the predictable result of testing one message at a time when the failures live between messages. The bugs that matter in conversation are cross-turn - the model forgets a constraint the user set three turns ago, loses the thread, or answers everything helpfully without ever closing the goal - and none of those show up in a per-reply score. Evaluating multi-turn conversations means scoring the dialogue as a whole. Here is how.

Why single-turn evals miss the real failures

A single-turn eval treats each reply as an isolated input-output pair. That works for a Q&A endpoint. It falls apart for a chatbot, because conversation quality is an emergent property. Consider a user who says “I’m vegetarian” early on, then ten turns later asks for dinner ideas and gets a steak recipe. Every individual reply is well-formed and fluent. The conversation is broken. The unit of evaluation for a chatbot is the conversation, not the message - and once you accept that, the metrics change.

The metrics that only make sense over a dialogue

Score at two levels. Turn level gives each reply a score so you can localize exactly where a conversation went off the rails. Conversation level gives the whole session one score for the outcome. The metrics worth tracking:

Metric	What it asks	Level
Context / knowledge retention	Does the agent remember earlier turns and stated constraints?	Conversation
Goal / task completion	Did the dialogue reach the user’s actual objective?	Conversation
Role adherence	Did the assistant stay in character and within policy throughout?	Conversation
Relevancy	Was each reply on-topic for the turn?	Turn
Faithfulness	Was each grounded reply true to its sources?	Turn

Retention and completion are the two that single-turn testing structurally cannot measure - they are defined by the relationship between turns. If you add nothing else to your eval when moving from Q&A to chat, add these two.

Step 1: Represent a conversation as an ordered set of turns

A multi-turn test case is not one input and one expected output. It is an ordered list of turns - user message, assistant reply, user message, and so on - carrying the state the conversation builds up. Illustrative shape:

# illustrative - structure, not a specific SDK
conversation = [
  {"role": "user", "content": "I'm vegetarian and allergic to nuts."},
  {"role": "assistant", "content": "..."},
  {"role": "user", "content": "Suggest a dinner."},
  {"role": "assistant", "content": "..."},   # scored: did it respect both constraints?
]

The constraints set in turn 1 are what turn 4 is graded against. That carried state is the whole point.

Step 2: Capture real conversations, then simulate the rest

Build your dataset from two sources. First, mine production - capture full sessions as traces in Langfuse, which groups turns into sessions so a conversation reads as one thread, and it is MIT-licensed and free to self-host. Label the good and bad sessions to form your eval dataset.

Real traces only cover paths users have already taken, so simulate the rest. DeepEval offers multi-turn conversation simulation among its metrics, and Maxim simulates realistic multi-turn users across thousands of scenarios and personas - a way to stress conversation paths you have never seen in production. Maxim’s honest trade-off is billing per seat and capping logs at once. Real conversations give you fidelity; simulated ones give you coverage - you want both.

Step 3: Score with a calibrated judge

Retention, completion and role adherence are judgments with no single correct string, so LLM-as-a-judge is the practical scorer. Give it the whole conversation plus a clear rubric - for goal completion, spell out what “the user’s goal was met” concretely means for your app. Two cautions specific to multi-turn:

• A whole conversation is a lot of tokens per judged metric, so multi-turn judge suites cost noticeably more than single-turn ones. Cost-model before you scale.
• Calibrate the judge against human-scored conversations before you trust it, exactly as in the LLM-as-a-judge guide. An uncalibrated judge on long inputs drifts faster, not slower.

For comparing agent versions across a conversation dataset cleanly, Braintrust runs custom scorers as plain functions and compares experiments side by side, with CI/CD quality gates that block a merge on a statistically significant regression. Its billing meter counts processed data in GB with no cap, and long conversations are large payloads - set alerts.

Step 4: Move a slice online

Offline conversation evals test the dialogues you thought to write. Production surfaces the ones you did not. Run online evaluation on a sample of live sessions and feed the failures back into your dataset, so tomorrow’s test set is built from today’s real breakdowns. The loop only compounds if live failures become labeled cases.

Common mistakes

• Scoring turns in isolation. The failures that matter are cross-turn - measure retention and completion at the conversation level.
• Only using real traces. They cannot cover unseen paths - simulate to get coverage on personas and edge dialogues.
• Trusting an uncalibrated judge on long inputs. Calibrate against human-scored conversations first.
• Ignoring the token cost. Whole-conversation judging is expensive - cost-model the suite before scaling it.

The bottom line

Evaluate the conversation, not the message. Capture real sessions in Langfuse, fill the gaps with simulated dialogues from DeepEval or Maxim, score retention and goal completion with a calibrated judge, compare versions in Braintrust, and move a slice online. For the chatbot tooling landscape, see the best LLM eval tools for chatbots, and for the broader agent workflow, how to evaluate AI agents.