How to Evaluate AutoGen Agents in 2026 - Multi-Turn Runs, Loop Convergence and Termination

Microsoft AutoGen frames a task as a conversation - agents send messages back and forth until a termination condition fires or a turn cap stops them. That conversational shape is the whole challenge of evaluating it. You are not scoring a single response, you are scoring a dialogue - whether it converged, whether it stopped for the right reason, and whether every turn actually earned its place. A run that produces a plausible last message but only got there by hitting the max-turns limit is a failure wearing a success mask. Here is how to evaluate AutoGen runs so you can tell the difference.

The four questions an AutoGen eval has to answer

A single-shot LLM eval asks “is this answer good.” A multi-turn AutoGen eval asks four questions at once:

• Did it converge? Did the conversation reach a conclusion, or just stop?
• Did it terminate cleanly? Did it stop on the intended termination condition, or did it hit the max-turns cap - which is a failure signal, not a graceful exit.
• Did each turn add progress? Or did agents re-cover ground, re-ask, or drift off task.
• Is the final output correct? Faithful to sources, and actually the thing the user asked for.

Hitting the turn limit is the single most under-counted AutoGen failure - the run does not crash, it just quietly caps out, and if you only score the last message you will call it a pass. Score termination explicitly.

Step 1: Capture the whole conversation as a trace

You cannot score a dialogue you did not record. Instrument AutoGen so the full multi-turn transcript lands as one trace, with a span per turn showing which agent spoke, what it said, and any tool calls. Langfuse is the open-source default - MIT-licensed, free to self-host, framework-agnostic, and it nests the turns under one trace so you can scrub the whole conversation. It also gives you datasets and human annotation queues for the calibration step. The operational catch is the four-service v3 self-host, so the free cloud tier is the shortcut if you would rather not run it. The trace is where you will read turn-by-turn progress and spot the run that looped without converging, the same way you would debug a LangGraph agent or any AI agent under test.

Step 2: Score convergence and termination first

Before quality, score whether the run behaved. Two checks, both cheap:

• Termination check - did the conversation end on your intended termination condition, or did it hit the turn cap? This can often be a deterministic assertion on the run’s stop reason, and deterministic checks never drift and cost nothing.
• Turn-count and cost - a run that converged in four turns and one that scraped in at nineteen are different risk profiles even if both “passed.” Track turn count and token cost as efficiency signals alongside quality.

A non-converging loop should surface as a failed metric, not as a surprise on your token bill. These structural checks are what separate multi-turn evaluation from single-shot scoring.

Step 3: Score turn-level progress and final quality

Now the model-graded part. For open-ended quality - did each turn stay on task and add progress, is the final output faithful and correct - you need LLM-as-judge with a rubric. DeepEval fits because it is framework-agnostic and ships conversational and multi-turn metrics alongside its 50-plus research-backed metrics including G-Eval, faithfulness and hallucination. You run the AutoGen conversation, capture the transcript, and assert metrics on the turns and the final output in a pytest-style suite you run in CI with deepeval test run. The gotcha - nearly all its metrics are LLM-as-judge, and a long transcript means many judge calls, so multi-turn suites compound API cost faster than single-shot ones. Cost-model before you scale.

And calibrate the judge before you trust it. Score a sample of conversations by hand in Langfuse’s annotation queues, tune the rubric until the judge agrees with the humans, and re-check periodically. The mechanics are in the LLM-as-a-judge guide and the broader how to evaluate AI agents walkthrough. An uncalibrated judge scoring a multi-turn run is a guess with more words in it.

Step 4: Run it many times and compare versions

Multi-agent conversations are non-deterministic, and multi-turn runs are more variable than single-shot calls - the same task can converge in three turns once and loop to the cap the next time. One pass is not a result - run each scenario several times and read the distribution of convergence, termination and quality. For comparing versions when you change an agent’s system message or the termination condition, Braintrust is the most turnkey - autoevals scorers, an experiments view that compares runs, and CI/CD quality gates that block a merge on a statistically significant regression. The billing gotcha matters here more than usual - Braintrust meters processed data in GB with no hard cap, and long multi-turn transcripts are among the most byte-heavy workloads there is, so set billing alerts on day one. This is the same regression discipline as how to run LLM regression tests.

The short version

1. Score four things - convergence, clean termination, turn-level progress, final correctness.
2. Capture the full transcript as a trace in Langfuse.
3. Check termination and turn count deterministically - hitting the cap is a failure.
4. Score quality with DeepEval’s multi-turn metrics and a calibrated judge.
5. Run many times and compare versions in Braintrust, watching the billing meter.

Evaluate the conversation, not the last message, and “the agent seems fine” turns into “it converges 60% of the time and caps out the rest” - a number you can act on. Every price and date here was read from each vendor’s own pages on 23 July 2026, and this category ships breaking changes monthly.