Evaluating AI Agents

By this point in the course you have learned to score single responses with metrics and LLM judges, and to build a repeatable eval pipeline. Agents break most of that. An agent does not return one answer to one prompt. It plans, calls tools, reads the results, decides what to do next, and loops until it thinks it is done. The thing you are evaluating is a trajectory, and a trajectory can be wrong in ways a final answer never reveals.

Why agents need a different eval approach

Consider a support agent asked to refund an order. The correct final message might be “Your refund of $40 has been processed.” That single sentence looks fine to a response-level check. But underneath, the agent might have looked up the wrong order, called the refund tool twice, or issued the refund before confirming the customer’s identity. The output passed. The behavior was dangerous.

This is the core problem: with agents, the final answer is a lossy summary of what actually happened. Two runs that produce identical text can have wildly different quality underneath. So agent evaluation splits into two questions that you score separately.

First, outcome quality: did the agent achieve the goal? Second, process quality: did it get there through a sound sequence of decisions? You need both. Optimizing only for outcomes rewards agents that get lucky. Optimizing only for process rewards agents that follow the rules and still fail.

The trajectory is the unit of evaluation

A trajectory is the full record of a single agent run: the user goal, each reasoning step, every tool call with its arguments, the tool response, and the final output. Everything you score hangs off this structure. The most useful trajectory-level checks are:

Tool call accuracy. Did the agent select the right tool for each step? A weather agent that answers a flight question by calling get_weather failed even if the text sounds confident. You assert on the tool name and often the arguments.

Parameter correctness. The agent called refund_order, but did it pass the right order_id? Wrong arguments to the right tool are one of the most common and most expensive agent failures.

Goal completion. Ignore the path for a moment - did the end state match what the user wanted? For a booking agent, was a booking actually created with the correct details?

Efficiency. How many steps did it take? An agent that reaches the goal in 12 tool calls when 3 would do is burning latency and tokens, and usually signals confused reasoning.

Safety and forbidden actions. Did the agent avoid steps it should never take, like deleting data or issuing a refund without verification? These are pass or fail, not scored on a curve.

Resist the urge to demand golden paths

A tempting design is to record one “perfect” trajectory per test case and require the agent to reproduce it step for step. This almost always backfires. There are many correct ways to answer a question, and a rigid golden path fails good agents for taking a different but valid route. Your eval becomes noise, and worse, it discourages the flexible reasoning agents are supposed to have.

Assert on invariants instead. Say what must be true (the search_flights tool was called before book_flight, the final booking matches the requested date) and what must never happen (no payment tool called with an unverified user). Leave the rest free. This is the difference between testing behavior and testing a transcript.

Make agent tests deterministic with record and replay

Agents that hit live APIs are miserable to test. Responses change, services rate-limit you, and every run costs money. The standard fix is record and replay. You run the agent once against real tools and capture every tool response into a fixture. On every subsequent test run, the agent’s tool calls are served from those fixtures instead of the network. The agent’s decisions are what you are testing, and those stay meaningful even with frozen tool outputs.

This also lets you inject failures deliberately. Replace a recorded success with a timeout or a malformed response and check that the agent recovers instead of hallucinating a result. Robustness to bad tool output is a distinct axis of agent quality that live testing rarely exercises on purpose.

How the real tools do it

Trajectory evaluation is where dedicated tooling earns its keep, because you cannot eyeball a hundred multi-step runs. LangSmith was built around LangChain and LangGraph agents and captures the full run tree, so you can inspect each step, attach evaluators to intermediate nodes, and score tool selection directly on the trace. Langfuse takes an open-source, framework-agnostic approach - nested spans model the agent loop, and you can run dataset experiments over recorded traces. Braintrust leans into the experiment-and-compare workflow, letting you diff two agent versions across the same task set. Maxim markets itself specifically around agent simulation and multi-turn testing, running an agent against simulated users to probe how it handles conversations rather than isolated prompts.

The practical pattern across all of them is the same: instrument the agent so every step becomes a span, build a dataset of goals with outcome assertions, run the agent over the dataset, and let the tool surface which trajectories regressed. For the deeper mechanics, our guide on testing AI agents walks through assertion design, and benchmarking AI agents covers comparing agent versions at scale. If your problem is production visibility rather than pre-release testing, the best AI agent observability tools maps the landscape, and the general framing in how to evaluate LLM applications still applies underneath.

A minimal agent eval, start to finish

Start with 20 to 30 real goals your agent should handle. For each, write down the invariants: which tools must be called, which arguments must be correct, what the end state must be, and what must never happen. Run the agent, capture trajectories, and score against those invariants rather than a golden path. When a case fails, look at the trajectory, not just the output, to see where the reasoning broke. Freeze tool responses so the suite is deterministic, then wire it into CI exactly like the regression tests from the previous chapter.

Once your agent works in tests, the next question is what it does with real users and real inputs you never anticipated. That is a production concern, and it is where we go next.

Key takeaways

• Agents produce trajectories, not answers - score both outcome quality and process quality.
• Assert on invariants (right tool, right arguments, goal reached, forbidden actions avoided), not on golden step-by-step paths.
• Use record and replay to make agent tests deterministic and to inject tool failures on purpose.
• Instrument every agent step as a span so tools like LangSmith, Langfuse, Braintrust, and Maxim can show you where a run went wrong.

Next up: Observability and Tracing in Production, where these trajectories become live traces from real traffic.