Agentic AI Evaluation Metrics: The Complete Reference

Agentic AI evaluation metrics score four things: what the agent did (outcome), how it called tools (action), the path it took (trajectory), and how a group of agents coordinated (system). DeepEval’s Tool Correctness computes correctly-used tools ÷ total tools called. Azure’s Intent Resolution scores the agent’s early actions against the user’s goal. NVIDIA names Task Success Rate, Tool Call Accuracy and Trajectory Efficiency as its core three. Below: 31 metrics with their documented computations, a crosswalk showing that four vendors’ “task completion” numbers are not the same number, a cost model for the judges themselves, and the run counts you need before a delta means anything.

A note on sourcing. Every definition is attributed to the vendor documentation it came from and dated to when that page was consulted (August 2026 unless stated). Most sources on this topic sell evaluation tooling, and are labelled as such. Nothing here is a measured benchmark. Where documentation is silent on how a metric is computed, this page says “not documented” rather than guessing.

Agentic AI evaluation metrics, in one paragraph

An agentic AI evaluation metric is a scoring function applied to a trace, not to a string. It measures reasoning steps, tool calls, the ordered path, and the state of the world after the run finished. Final-text scoring covers a fraction of what can go wrong.

Every metric below carries one of four scopes:

Scope	Attaches to	Question it answers	Example metrics
Component	One span: a tool call, a handoff, a model turn	Was this action correct?	Tool Correctness, Argument Correctness, Handoff Correctness, schema compliance
Trajectory	The full ordered run	Was the path sensible and efficient?	Plan Adherence, Step Efficiency, loop detection
Outcome	End state, including systems outside the agent	Did the world change correctly?	Task Success Rate, side-effect assertion, false-completion rate
System	Interactions across agents and sessions	Did coordination hold?	Invariant violation rate, ping-pong handoff rate, memory poisoning tests

Model benchmarks answer a different question. MMLU, GSM8K and HumanEval score the engine on static prompt-in/response-out pairs; NVIDIA separates these from agent benchmarks like GAIA, SWE-bench, WebArena, τ-bench and AgentBench (NVIDIA Developer Blog, vendor source). A strong benchmark score is a prerequisite for reliability. It predicts almost nothing about whether your refund agent issues one refund or two.

New to the architecture underneath? Start with agentic AI system design, then come back for the scoring layer.

Why model metrics don’t transfer: four gaps agentic AI evaluation metrics close

Many correct answers, no reference string. BLEU and ROUGE assume one. An agent told to “find the cheapest flight and hold it” has dozens of acceptable action sequences and one acceptable end state.

Success is often a side effect with no text attached. The refund landed. The ticket got assigned. None of that appears in the final message, and the final message can describe it perfectly while none of it happened.

Identical answers, wildly different paths. Three tool calls or forty. Eight hundred tokens or eighty thousand. Only trajectory-scoped metrics tell them apart.

One run proves nothing. Temperature, tool-output variation, retrieval ranking and upstream latency all move between runs.

The failure class that makes this urgent is what Confident AI calls false task completion, or ghost actions: the agent asserts a booking was made and the trace shows the booking tool was never invoked. Output-only scoring rates it highly, which is why their guide calls it the most dangerous agent failure mode (Confident AI, vendor source). A support agent hits a 403 on the refund API, swallows the error, and replies “I’ve processed your $89 refund, you’ll see it in 3-5 business days.” Fluent. Polite. On-intent. A final-answer judge gives it a 0.9.

So 17 of the 31 metrics below are trajectory- or outcome-scoped, and the highest-value one on this page is a post-run assertion against your own database.

Practitioners keep asking the operational version. An Ask HN thread from a developer building coding agents asks what the actual standard is for running agent evals in practice, noting that task definitions differ dramatically across domains (news.ycombinator.com/item?id=45988855). Vendor lists answer “what could you measure.” They rarely answer “what do you run on Tuesday.”

The catalogue: 31 agentic AI evaluation metrics by scope

Each entry gives the computation, the failure it catches, whether it needs ground truth, and whether it is deterministic or LLM-judged. That last split sets both the cost of running the metric and whether the metric has an error rate of its own. IBM’s function-calling section splits on the same line (IBM Think, vendor source).

Run every deterministic check before any judge fires. A wrong function name costs nothing to detect.

Outcome scope

1. Task Success Rate (TSR). Share of goldens where the agent satisfied intent and constraints. NVIDIA’s framing is the useful part: the task is not “update this record” but “update this record through this API within two tool calls.” Needs ground truth. Deterministic when success is checkable.

Binary scoring is harsher and more honest. Partial credit lets a regression hide: an agent that used to complete 90 goldens fully and now completes 60 fully plus 40 halfway shows a higher mean while being worse at the job. Report the fully-complete count alongside any graded score.

2. False-completion rate. Share of runs where the agent claimed success and the outcome assertion failed. Not 1 − TSR. A run that fails and says so is operationally fine; the user retries. A run that fails and claims success is a chargeback.

3. Error rate. Share of runs ending in an unhandled exception, an unrecovered tool error, or a step-limit kill. Free, and the first thing to check when TSR drops.

4. Ground-truth side-effect verification. After the run, query the system: does the refunds table hold exactly one row for this order, amount 89.00, created inside the run window? Implement as post-run assertions against a seeded environment you can roll back. This is the only hard defence against ghost actions. A judge reading a transcript can be persuaded. SELECT count(*) FROM refunds WHERE order_id = ? cannot.

5. Task Adherence (Azure AI Evaluation). Final response scored against the original request (Microsoft Tech Community, vendor source, updated 2025-04-14). LLM-judged, referenceless. The Azure AI Foundry evaluator library has grown past that post’s three metrics.

6. Intent Resolution (Azure). Whether the agent’s initial actions reflect correct understanding of the goal, on a 1-to-5 scale with a documented default pass threshold of 3 (same source).

Read 5 and 6 together. Low intent resolution with high task adherence means the agent answered the wrong question beautifully. The inverse means it understood and fell over during execution. Prompt and routing work versus tool reliability work.

7. TaskCompletionMetric (DeepEval). Documented as AlignmentScore(Task, Outcome), with both inferred from the trace rather than supplied as a label. Examples initialise it with threshold=0.7 and model="gpt-4o" (DeepEval docs, vendor source). Referenceless means you can run it on production traffic with no labels. It also means the metric infers what the task was from the trace it is grading, so an agent that quietly redefines the task scores well.

Action scope: tool and function calling

8. Tool Correctness (DeepEval). Correctly used tools ÷ total tools called, with five configurable strictness modes. Deterministic; needs an expected tool list.

Strictness mode	What it compares	What the looser mode misses
Tool-name matching (default)	Called names vs expected	Wrong tool selected entirely
Input-parameter matching	Names and arguments	Right tool, wrong order ID
Output matching	Names and returned outputs	Tool returned an error the agent ignored
Ordering	Sequence of calls	Refund issued before eligibility check
Exact matching	Exact set, no extras	Exploratory calls padding the run

Ordering strictness is under-used. Financial and permissions workflows have a required order, and a correct-set-wrong-order run is a compliance failure that name matching scores 1.0.

9. Argument Correctness (DeepEval). Correctly generated parameters ÷ total tool calls, documented as fully LLM-based and referenceless: arguments are judged against input context, not expected values. So 0.94 here is really 0.94 plus or minus a judge error nobody has measured for your domain.

10-14. IBM’s rule-based function-calling taxonomy. Five deterministic checks cheap enough to run on every span in CI:

• Wrong function name. get_order_status when the schema exposes fetch_order_status. Catches stale names after a schema change.
• Missing required parameter. issue_refund(order_id=...) with no amount.
• Wrong value type. "89.00" where the schema declares a float.
• Value outside allowed set. status="pending_review" where the enum is {open, closed, escalated}.
• Hallucinated parameter. Invents priority="high" on a tool with no such field.

15. Parameter value grounding (IBM, LLM-judged). Every argument must trace to user text, a prior tool output, or a documented default. An argument from nowhere is a fabricated fact with side effects.

16. Unit transformation correctness (IBM, LLM-judged). Dates and currency dominate here. A user says “next Friday” and the agent writes 2026-08-21 when the correct date is 2026-08-28. A user says “£40” and the tool expects minor units, so 40 books forty pence.

17. Tool selection precision and recall. The golden requires lookup_order, check_refund_eligibility, issue_refund. The agent calls those plus search_kb. Precision 3/4 = 0.75, recall 3/3 = 1.00: one wasteful lookup. Had it skipped the eligibility check, precision would read 1.00 and recall 0.67. The more dangerous run scores perfectly on precision alone. Report both.

18. Schema compliance rate. Share of tool calls validating against the declared JSON schema on first attempt. The cleanest early warning after a model version change.

19. Retry-on-invalid-call rate. Share of calls needing at least one repair. A model that fails then self-corrects shows high final compliance and high latency.

20. Handoff Correctness. Did the router send the task to the right agent (Confident AI, vendor source). In LangGraph and CrewAI this maps to an edge assertion; in the OpenAI Agents SDK, to the handoff object the orchestrator returns.

Trajectory scope

21. Plan Quality (DeepEval). AlignmentScore(Task, Plan), judged before execution.

22. Plan Adherence (DeepEval). AlignmentScore((Task, Plan), Execution Steps): did the agent do what it said it would.

Both need an explicit plan artifact. A ReAct-style agent reasoning inline gives these metrics nothing to grade, and you get scores that look like numbers and mean nothing.

23. Step Efficiency (DeepEval) / Trajectory Efficiency (NVIDIA). Steps and tokens per task. Compute it over successful tasks only. Averaged across all runs it rewards fast failures, and the agent that gives up in two steps becomes your most efficient agent.

24. Steps to first meaningful tool call and 25. tokens before first real side effect. Confident AI names both as leading indicators of reasoning thrash. An agent burning 4,000 tokens before touching a tool is one prompt regression from a timeout, and TSR won’t move until it happens.

26. Loop and repetition detection. Flag duplicate calls with near-identical arguments (normalise whitespace and key order before hashing), repeating span patterns, and any run hitting the step ceiling. The ceiling is a kill switch. Set it, then count how often it fires.

27. Reasoning soundness. Sample traces on a rotating slice and label each sound, partially flawed, or incorrect. Not a per-run gate. It is how you catch right answers reached through wrong logic.

28. Intermediate-output groundedness. Score the tool results feeding the answer. A search tool returns three stale documents and the agent writes a polished, entirely wrong summary. Answer-level faithfulness scores it as faithful, because it is faithful to bad context. RAGAS context precision and recall port cleanly here.

System scope: multi-agent

29. Invariant violation rate. Invariants pass or fail, and a failure blocks the build. QuantumBlack lists the recurring multi-agent failure modes they guard against: oscillating handoffs, deadlocks with unowned tasks, conflicting writes such as double refunds, memory poisoning, and resource-exhaustion cascades (QuantumBlack on Medium, consultancy source; the case examples are anonymised and unverifiable, so treat the concepts as the contribution).

Starter set for a support agent with refund powers:

• No order receives two refund writes in one session.
• No ticket ends a run without an owner.
• No account balance goes negative.
• No PII appears in log or trace payloads.
• Nothing outside the read-only allow-list runs before human confirmation.

These are unit-test assertions for a system whose control flow you don’t fully determine.

30. Coordination metrics. Handoffs per task, duplicate work rate, deadlock rate, oscillating-handoff rate (A → B → A → B), time to resolution, resource footprint per task. All deterministic if traces carry agent identity on every span. AutoGen, LangGraph and CrewAI emit it by default; a hand-rolled orchestrator usually doesn’t.

31. Memory-layer metrics: stale recall, state drift, memory poisoning. A concrete eval: in turn 2 of a scripted session, have the simulated user assert a false fact (“my account is on the enterprise plan”); in turn 14, ask a question whose answer depends on the true plan; assert that the agent re-reads the system of record. Score the share of poisoning attempts that survive to influence a later answer. Stale recall is the same test with a fact that has since changed.

The better-than-its-parts test. Run the same golden set against one agent holding all the tools, then against your topology. Compare cost per successful task, wall-clock time and TSR. If the topology wins on none of them, collapse it.

Safety, policy and adversarial metrics

Prompt injection success rate splits into two numbers. Direct injection is a hostile user typing “ignore previous instructions.” Indirect injection arrives through content the agent reads: a poisoned document, a ticket body, a web page, a tool response. OWASP’s LLM Top 10 lists prompt injection as LLM01 and excessive agency as LLM06, which gives your risk register a shared vocabulary. The indirect case scales with every data source you connect. Build a corpus of poisoned artifacts and measure the share that causes a tool call the user never asked for.

Policy adherence rate. Share of responses compliant with written, machine-checkable rules. “Be helpful and appropriate” is not a metric. “Never quote a refund above the order total” is. The NIST AI Risk Management Framework’s Measure function and the EU AI Act’s Article 15 accuracy-and-robustness requirement are the usual anchors for this row.

Excessive-agency counters. Unauthorised tool invocations, actions outside the allow-list, writes without confirmation. These sit at zero and page someone when they don’t.

Bias scoring across cohorts: run matched goldens differing only in a protected attribute or dialect, then compare outcome rates. Escalation rate works as a robustness proxy, with the caveat that it moves for good reasons too. Guardrail hit rate is a runtime counter: offline you ask whether the agent ever does this, online you ask whether today’s rate changed. Our AI guardrails implementation guide covers enforcement.

Cost, latency and the operating envelope

Cost per successful task, not cost per run. An agent at $0.20 per run with a 50% success rate costs $0.40 per successful task. An agent at $0.32 per run succeeding 88% of the time costs $0.36. The cheaper-per-run agent is the more expensive agent, and cost-per-run dashboards say the opposite every day.

Split model latency from workflow latency. Model latency is time to first token and tokens per second. Workflow latency is serial tool calls, extra round-trips, polling loops, queue waits. Different fixes entirely: routing versus concurrency design.

Budgets as constraints, not reports. NVIDIA recommends expressing these as explicit budgets, such as the 95th percentile of tasks landing under N tokens and M tool calls. A percentile constraint is enforceable in CI. A mean in a dashboard gets read after the incident.

Retry amplification. Track retries per task and tokens spent on retried work separately. A flaky endpoint multiplies spend while TSR, tool correctness and adherence stay green.

Voice agents need their own envelope. A team running LiveKit voice agents describes having “no way to see TTFT (time to first token) per call,” difficulty measuring cost per call, and no visibility into latency across the STT → LLM → TTS chain (news.ycombinator.com/item?id=44866675). That is an operator’s account from a founder building tooling, not an independent measurement. The metrics it points at: barge-in handling rate, turn-taking latency, and the share of calls where one stage blew its budget.

Crosswalk: agentic AI evaluation metrics under five vendor names

Two dashboards reading 0.82 “task completion” are not showing the same thing. Cells are filled only from vendor documentation; “not documented” is itself a finding when you choose a tool.

Construct	Azure AI Evaluation	DeepEval	NVIDIA NeMo Agent Toolkit	IBM watsonx	Arize Phoenix
Task outcome	Task Adherence, response vs request, LLM-judged, 1-5	TaskCompletionMetric, AlignmentScore(Task, Outcome), trace-inferred, threshold 0.7	Task Success Rate, computation not documented	Success rate, named in watsonx.governance, computation not documented	Prompt-template eval; Phoenix supplies the rubric, you supply the judge
Intent understanding	Intent Resolution, initial actions, 1-5, default pass 3	Partially covered by Plan Quality	Not documented	Not documented	Covered by the agent-goal rubric
Tool selection	Tool Call Accuracy	Tool Correctness, 5 strictness modes	Tool Call Accuracy, computation not documented	Rule-based taxonomy of five error types	Tool-calling template, judged against your tool definitions
Argument correctness	Folded into Tool Call Accuracy	Argument Correctness, LLM-judged, referenceless	Not documented	Parameter grounding + unit transformation	Folded into the tool-calling template
Plan adherence	Not documented	Plan Adherence, AlignmentScore((Task, Plan), Steps)	Not documented	Not documented	Build it on the span-level eval API
Step efficiency	Not documented	Step Efficiency, steps/tokens per task	Trajectory Efficiency, computation not documented	Not documented	Convergence eval: steps vs shortest observed path
Safety / policy	Responsible AI set: violence, self-harm, hate, protected material, indirect attack	Guardrail and red-team metrics, shipped separately	Not documented in this post	Responsible-AI metric block	Hallucination, toxicity, QA-correctness templates

Sources consulted August 2026. MLflow, Weights & Biases Weave, Langfuse and Braintrust ship their own agent evaluators on top of the same trace data, so the naming problem multiplies as you add platforms.

Where dashboards get misread:

• Intent Resolution and Task Adherence measure different halves of the run. Averaging them into one “quality” tile destroys the diagnosis.
• DeepEval’s TaskCompletion is referenceless. It can score 0.9 on a run where the agent solved a task adjacent to yours.
• Scale mismatch breaks naive averaging. Azure returns 1-5. DeepEval returns 0-1. Averaging a 4 and a 0.8 into “2.4 quality” happens more than anyone admits.
• Reference requirement drives your dataset budget. Referenceless metrics run on unlabelled traffic. Tool Correctness, side-effect assertions and Handoff Correctness need a labelled expected value per golden, and labelling is where the human hours go.

How far these implementations diverge on an identical trace set is unanswered. It needs a funded side-by-side run with a shared corpus, which no public source has published.

Still choosing? Our multi-agent framework comparison covers which evaluators each framework integrates out of the box.

What agentic AI evaluation metrics cost to run

Judge cost per trace = (trace tokens + rubric tokens) × input price + verdict tokens × output price. Total = that, × judged metrics × goldens × runs per golden.

Trajectory-scoped metrics re-read the entire trace, so judge cost scales with the quantity an inefficient agent inflates. An agent thrashing to 60,000 tokens costs roughly 8× more to judge than one resolving in 7,500, per metric, per run. Your eval bill grows fastest on the runs you most want to catch.

Worked scenario at GPT-4o-class pricing, read 2026-08-19 at $2.50 per million input tokens and $10.00 per million output. Assume 12,000 trace tokens, 800 rubric tokens, 250 verdict tokens, four judged metrics, 200 goldens, one run each: (12,800 × $2.50/1M) + (250 × $10/1M) = $0.0345 per metric per trace, or $27.60 for the pass. Five runs per golden: $138. Sampling 1% of 100,000 production traces on the same four metrics: $138 again.

Mitigations, in payoff order:

1. Deterministic checks first. IBM’s five-item taxonomy costs CPU time. Every failure it catches is a judge invocation you never pay for.
2. Cheap model for high-volume metrics. At GPT-4o-mini pricing of $0.15 per million input tokens, that 200-golden pass costs $1.86 instead of $27.60.
3. Expensive judge on sampled or failing traces only.
4. Prompt-cache the rubric. OpenAI discounts cached input by 50%, Anthropic by 90% against a 25% write premium.
5. Truncate deliberately. Judging first and last N spans plus a summary changes what the metric measures. Document it, because a truncated trajectory metric is a different metric.

This is arithmetic over published price lists, not a bill anyone paid.

How many runs before a difference in your agentic AI evaluation metrics is real

Task success on a golden is a Bernoulli trial. With 50 goldens and one run each, 82% to 86% is two extra goldens passing, and a 95% confidence interval on 50 binary trials runs roughly ±10 points. Noise. Teams ship on it constantly.

Report an interval. The normal-approximation half-width is 1.96 × √(p(1−p)/n). At p = 0.85 and n = 50 that’s ±0.099. At n = 200, ±0.049. At n = 500, ±0.031. Detecting a 5-point regression takes a few hundred goldens. Above p = 0.9, switch to Wilson score intervals, where the normal approximation starts producing upper bounds above 1.0.

Repeat each golden k times. Fifty goldens once and fifty goldens five times give the same coverage and completely different information. Repetition separates scenario difficulty (fails every time) from run-to-run variance (passes 3 in 5). Capability gap versus reliability gap. Mean pass rate blends them into one uninformative number.

Use pass^k for anything touching money. Pass^k is the share of goldens where all k runs succeeded. τ-bench popularised it, and its published numbers make the case: a model near 0.6 pass^1 on the retail split drops to roughly 0.25 at pass^8. “Succeeds 4 times out of 5” is not a passing grade when the fifth run is a real customer.

Flakiness rate deserves its own line. Share of goldens whose outcome is inconsistent across identical runs. A rising flakiness rate is a real regression even when mean TSR is flat. Usually a tool got slower, an index changed, or a prompt edit pushed the model toward a decision boundary.

Measure the judge’s variance too. A judge that disagrees with itself 12% of the time cannot detect a 5-point regression at any dataset size.

Setting temperature to 0 does not make an agent deterministic. Tool outputs vary, retrieval rankings shift, timestamps move, upstream APIs return different payloads. Temperature is one source of variance out of five.

Choosing your five to seven

QuantumBlack reports teams settling on five to seven metrics per workflow, spanning capability, robustness, safety, human interaction and economics. Define them alongside the workflow, then hold them stable long enough to see a trend.

Agent shape	Mandatory	Recommended	Noise for this shape	Required invariant
Single-turn tool user	TSR, Tool Correctness (input-param), side-effect assertion	Schema compliance, cost per successful task	Conversation completeness, handoff metrics	No write outside the allow-list
Multi-turn conversational	TSR, Intent Resolution, conversation completeness	Turn relevancy, escalation rate	Plan adherence unless it plans explicitly	No PII echoed into the transcript
Multi-agent	Invariant violation rate, Handoff Correctness, cost per successful task	Ping-pong rate, duplicate work rate	Single-agent step efficiency in isolation	No conflicting write to one record per session
Long-horizon stateful	Checkpoint success rate, state consistency at resume, cost per unit of progress	Stale recall, poisoning survival rate	Single-run TSR as headline	Resume state matches checkpoint on critical fields
Voice / realtime	Per-call TTFT, task success rate, cost per call	Stage-decomposed latency, barge-in rate	Token-level trajectory efficiency	No call exceeds the latency ceiling without fallback

Whatever the shape, keep one outcome metric, one action metric, one cost metric, one safety assertion. A 0.87 “agent quality” number tells you nothing about which of four unrelated systems broke.

Goodhart, with specifics. Optimising step efficiency alone produces agents that skip verification, because verification is pure overhead in that objective. Optimising task completion alone produces agents that claim success, because claiming is cheaper than achieving. Optimising cost alone produces agents that stop trying. Pair every efficiency metric with a correctness metric and every correctness metric with a cost metric.

Thresholds and CI gates

Three gate types, and conflating them is why most eval configs are unusable.

Hard invariants block the build on any violation. Zero double refunds is not 99.9% of runs without a double refund.

Regression gates cap the drop versus the baseline commit. This catches model upgrades and prompt edits without you knowing what “good” means in absolute terms.

Absolute floors apply to a small critical-path subset you understand deeply.

gates:
  - metric: invariant.no_double_refund
    type: hard
    threshold: 0
    action: block
  - metric: side_effect_assertion.pass_rate
    type: absolute_floor
    threshold: 1.00
    subset: critical_path   # 24 goldens
    runs_per_golden: 5
    aggregation: pass^k
    action: block
  - metric: task_success_rate
    type: regression
    max_drop_points: 3
    baseline: last_green_main
    subset: all             # 200 goldens
    runs_per_golden: 3
    action: block
  - metric: cost_per_successful_task
    type: regression
    max_increase_pct: 25
    action: warn

Wire it into whatever runner you already use. The gate has to fail the build, not post a comment.

Numbers circulating in this space are frequently unsourced. One competing page states goals such as a 95% task-success rate, hallucination under 1%, and a two-second maximum response time, with no provenance (Accelirate). A high TSR on easy goldens is a weaker system than 80% on adversarial ones. Your threshold is defensible only against your own baseline.

When a component changes, whether a model upgrade, a prompt edit, a new tool or a schema change, re-run the full golden set against the prior baseline rather than the floor. A floor hides a 4-point drop that started 4 points above it.

Escalation policy. Invariant violations and side-effect failures page a human. Regression failures block the merge. Cost warnings appear in the weekly review. Write it down before the first incident.

Building the golden dataset the metrics run on

A golden is four things: input, available tools, expected outcome, constraints.

{
  "id": "refund-partial-shipped-order-014",
  "setup": {
    "conversation": [{"role": "user", "content": "I want to return the blender from order 44821, keeping the rest."}],
    "seed_state": {"order_44821": {"status": "shipped", "total": 213.40,
      "items": [{"sku": "BLND-9", "price": 89.00}, {"sku": "MUG-2", "price": 124.40}]}}
  },
  "available_tools": ["lookup_order", "check_refund_eligibility", "issue_refund", "search_kb"],
  "expected_outcome": {
    "side_effects": [{"table": "refunds", "count": 1, "order_id": 44821, "amount": 89.00}],
    "required_tools": ["lookup_order", "check_refund_eligibility", "issue_refund"],
    "forbidden_tools": []
  },
  "constraints": {"max_tool_calls": 6, "max_tokens": 12000, "must_not_refund_above": 89.00},
  "distribution": "in_distribution",
  "provenance": "production_failure_2026-06-11",
  "tier": "critical_path"
}

Ranked by yield: failed production traces first, then near-misses (completed with a retry or an odd step count), then edge cases from support tickets, then adversarial inputs including injection artifacts, then synthetic generation last. Synthetic-first datasets miss the out-of-distribution cases that dominate production failure, because a model writing test cases samples from the distribution the agent already handles. Track OOD share as an explicit property and set a floor around 20-30%.

Maintenance loop. Every incident becomes a golden that week. Every golden that has passed 50 consecutive runs across two model versions drops to a nightly tier. Otherwise CI time grows until someone disables the gate.

Standardise the trace schema. Align on the OpenTelemetry GenAI semantic conventions so your metrics survive a tooling change. Those conventions remain formally Experimental and attribute names have been renamed across releases, so pin the version in your instrumentation config. OpenInference (the Arize-backed spec Phoenix uses) and OpenLLMetry both map onto it, and LangSmith, Langfuse and Braintrust ingest OTel spans.

Public benchmarks supplement, never substitute. Domain-specific datasets are appearing: a Hacker News post describes a public insurance-agent benchmark of 510 scenarios across 10 categories and 9 insurance lines, with 357/76/77 train/val/test splits (news.ycombinator.com/item?id=46953463). That is one poster’s description of their own dataset; the Hugging Face dataset card governs.

For intake, see our production agent monitoring playbook.

Judging the judge

An LLM-judged metric has its own accuracy, and almost nobody measures it. You are grading a stochastic system with a stochastic grader and reporting to three decimals.

Calibration. Keep a versioned gold slice of 50 to 100 human-labelled traces. Report judge-vs-human agreement as Cohen’s kappa, which is the honest statistic when your labels skew 90/10 and always-say-pass scores 90% raw agreement. Above 0.6 is substantial. Under 0.4 is a judge you cannot gate on. When judge and human diverge, fix the rubric first: usually the metric was under-specified and the two graders answered different questions.

Pathologies to test by name:

• Position bias. In pairwise comparison the judge favours whichever response comes first. Run every pair twice with the order swapped. The MT-Bench authors documented this alongside verbosity and self-enhancement bias (arXiv 2306.05685).
• Length bias. Test with matched pairs differing only in verbosity.
• Self-preference. A judge sharing a base model with the agent rates it higher. Use a different model family on any release gate.
• Leniency drift. The provider updates the model behind your alias and every historical score becomes incomparable.

That last one has a mechanical fix. Pin the judge to an explicit version and record it with every score. gpt-4o is an alias. gpt-4o-2024-11-20 is a pin. Our LLM as judge guide covers rubric design.

Agent-as-a-judge, giving the judge tool access so it can inspect state, earns its cost on high-stakes trajectory scoring, where reading the database beats reading the agent’s claim about the database. A judge with tools also inherits the ghost-action problem it was hired to catch.

Humans stay irreplaceable on tone, policy nuance, and metric-green/user-red cases. Specify a sampling rate and a two-page rubric. “Add human review” without both quietly becomes zero reviews by week three.

From offline agentic AI evaluation metrics to production monitoring

Production removes your ground truth. Referenceless metrics, guardrail counters, invariant assertions and sampled judging survive. Anything needing an expected value does not, which is most of your CI suite.

Track drift per metric, not per system: score drift, input distribution drift, tool-error-rate drift, cost-per-task drift.

Outlier mining closes the loop. Route low-scoring production traces into the golden-set intake queue. It is the only mechanism keeping a dataset representative as users change.

Long-horizon agents are the weak spot. A founder building an ML-engineering agent frames existing tooling as something that “works for short, linear tasks, but falls apart once workflows become long-running, stateful, and feedback-driven” (news.ycombinator.com/item?id=46724298). That is a vendor’s framing of a problem their product addresses, matching a second practitioner account of multi-agent debugging pain (news.ycombinator.com/item?id=42293942, also a vendor).

Three metrics partially close it. Checkpoint success rate, scored per checkpoint: a 40-step workflow failing at step 38 is not the same result as one failing at step 3, and binary TSR treats them identically. State-consistency checks at each resume point, asserting the restored state matches the checkpoint on fields that matter. Cost per unit of progress, spend divided by checkpoints cleared, which catches an agent burning budget while stuck at step 12.

A one-page scorecard you can copy

Worked example: a support agent with lookup_order, check_refund_eligibility, issue_refund, search_kb.

#	Metric	Scope	Threshold	Gate type	Owner	Cadence
1	No double refund per session	System	0 violations	Hard invariant, blocks	Payments eng	Every commit
2	Side-effect assertion pass rate	Outcome	1.00 on 24 critical-path goldens, pass^5	Floor, blocks	Team lead	Every commit
3	Task Success Rate (200 goldens, 3 runs)	Outcome	No drop > 3 pts	Regression, blocks	Team lead	Every commit
4	Tool Correctness, input-param strictness	Component	No drop > 2 pts	Regression, blocks	Team lead	Every commit
5	Cost per successful task	Outcome	No increase > 25%	Regression, warns	Eng manager	Weekly
6	Flakiness rate	Outcome	≤ 5% inconsistent	Floor, warns	Team lead	Weekly

agent: <name>
shape: <single-turn | multi-turn | multi-agent | long-horizon | voice>
golden_set: {size: , critical_path_subset: , ood_share: }
judge: {model: , version_pinned: , calibrated_against: , agreement: }
metrics:
  - {name: , scope: , computation_source: , threshold: , gate: , owner: , cadence: }
review:
  daily:   [guardrail counters, invariant violations, error rate]
  weekly:  [cost per successful task, flakiness rate, drift per metric]
  release: [full golden set vs baseline, pass^k on critical path, judge recalibration]

Frequently Asked Questions

How do you measure agentic AI performance?

At four scopes. Outcome: task success plus an assertion against real system state. Action: tool selection, argument correctness, schema compliance. Trajectory: steps and tokens per successful task, plan adherence. Envelope: cost per successful task, latency, safety violations. A wrong tool argument, an inefficient path and a fabricated completion claim need different fixes and different owners, so one composite score hides all four.

What are evals in agentic AI?

A repeatable test: a golden run against the agent, with scoring functions applied to the resulting trace. Some are deterministic assertions on tool calls and end state. Others are LLM-judged, which handles nuance and carries an error rate you have to calibrate. Evals run locally, in CI as a merge gate, and on sampled production traffic.

What metrics evaluate AI models versus AI agents?

Model evaluation uses static benchmarks (MMLU, GSM8K, HumanEval) and text-quality metrics on prompt-in/response-out pairs. Agentic AI evaluation metrics add action and trajectory scopes, because agent success is often a side effect in an external system rather than text. NVIDIA draws the same line between model benchmarks and agent benchmarks such as GAIA, SWE-bench and WebArena.

How is agentic AI tested?

Define success as intent plus constraints. Build a golden set weighted toward production failures and out-of-distribution cases. Instrument full traces. Run deterministic assertions first, judges second. Repeat each critical golden k times. Gate CI on invariants and regressions, then feed low-scoring production traces back into the set. Cheap checks before expensive ones, repetition before any release decision.

What is a good task completion rate for an AI agent?

There is no universal number, because task success is a function of golden-set difficulty. τ-bench retail scores near 0.6 at pass^1 and 0.25 at pass^8 for a frontier model make a better calibration anchor than any round target. Targets copied between blog posts arrive with no provenance and no dataset description. Measure against your own baseline commit, report an interval, and use pass^k on flows that move money.

How much does LLM-as-a-judge evaluation cost on agent traces?

Use the formula: (trace tokens + rubric tokens) × input price + verdict tokens × output price, multiplied by metrics × goldens × runs. At GPT-4o-class pricing with 12,000-token traces, 200 goldens across four judged metrics runs about $27.60 per pass, or $1.86 on a mini-class judge. Trajectory metrics re-read the whole trace, so cost scales with trace length.

Is LLM-as-a-judge reliable enough to gate a release?

Only after calibration. Keep a human-labelled slice, report kappa above 0.6 before gating, pin the model version alongside every score, and test for position bias, length bias and self-preference. Deterministic assertions for hard gates. Judges for regression trends and triage.

Limitations of this page

The crosswalk derives from vendor documentation, not from running the frameworks. The cost model is arithmetic over price lists read on 2026-08-19. Several Hacker News citations are founders describing problems their own products address, labelled at the point of use.

One question this page deliberately leaves open: how far judge scores from Azure AI Evaluation, DeepEval and Arize Phoenix diverge on an identical trace set. That needs a funded study with a fixed judge model and a shared corpus. Estimating it would be inventing a result.

The most useful thing you can do this week costs nothing. Write five invariants for your agent, turn them into post-run assertions against your real database, and run them on every commit. Those assertions catch the failure class that every score-shaped metric on this page will happily rate at 0.9, and no dashboard of agentic AI evaluation metrics will tell you they fired unless you build the assertion first.