Evaluating Tool Calls Means Scoring Five Decisions, Not One

Evaluating tool calls means scoring five separate decisions. Did the agent call a tool at all when it should have? Did it pick the right function? Were the arguments schema-valid and semantically right? Was the call in the right position in the sequence? Did it handle the tool’s response, including errors? Start with two deterministic code checks against the recorded tool_calls field: tool-selection F1 and JSON-schema validity of arguments. Add required-argument matching, order scoring and an unnecessary-call rate once selection is stable. Reach for an LLM judge only where code cannot decide.

Last updated: 19 August 2026. Vendor capabilities in this category change monthly, so re-read the linked primary sources before you act on any row.

What “evaluating tool calls” actually means

The object under test is a tuple: decision-to-call, tool name, arguments, position in sequence, response handling. Score only the function name and you hide four of the five failure surfaces. Those four are where production agents break.

Attach each score to the right unit. Argument validity and tool-name correctness belong on the tool-call observation, the span that recorded the call. Task completion belongs on the root observation of the trace. Conversation-level outcomes belong on the session. Langfuse’s agent-evaluation guide documents this three-level split and describes trace-level LLM-judge evaluators as legacy in its product, with observation-level evaluators as the recommended shape.

Evaluating tool calls is not output evaluation. A model can write a flawless, well-cited paragraph about your refund policy and never call get_order_status. Every output metric you own will score it 1.0.

You will run these checks in two places. Offline, against a fixed dataset, on every pull request. Online, against sampled production traffic, continuously. Those two contexts want different metric mixes and have different cost drivers, both covered further down.

For readers who arrived looking for a spreadsheet: the ground-truth section gives a flat CSV and JSONL column schema you can paste into Sheets and fill in. It includes the two columns most eval-set formats forget.

One tool call exploded into its five decision points, each labelled with the metric that catches it and a code or judge badge. Built from the metric catalogue below.

The five ways a tool call goes wrong

1. Wrong decision. The agent calls a tool when it could have answered directly, or answers directly when it needed live data. When2Tool argues that existing tool-use benchmarks assume every task requires a tool. Nothing in the standard metric set penalises an agent that reaches for a function on every turn.

2. Wrong tool. Correct intent, wrong function. LangWatch’s tool-selection cookbook shows the asymmetry: send_email recall at 0.33 and create_reminder at 0.67, while get_calendar_events reached 1.00. Read tools get called. Action tools get skipped.

3. Wrong arguments. Malformed JSON, hallucinated field names, wrong types, missing required fields, extra parameters. There is a scoring trap here. Maxim’s guide illustrates it with a call of {"order_id":"12345","include_history":false} scored against an expected {"order_id":"12345"}. That call is correct. Exact-match scoring records it as a failure.

4. Wrong sequence. Right tools, wrong order. Or the same failing call repeated four times until the step budget runs out. Multi-step tasks need an ordered comparison. A set comparison scores a retry loop exactly like a clean run.

5. Wrong response handling. The tool returned a 500, or an empty result set, and the agent carried on as though it had data. T-Eval isolates this as a distinct “review” ability with five outcome labels: Success, Internal Error, Input Error, Irrelevant Response, Unable to Accomplish.

These fail independently. That is the whole point. An aggregate “tool accuracy” of 0.72 cannot tell you whether you have a naming problem, a schema problem or a retry loop. Build one score per failure class from day one. Retrofitting the split after six months of dashboard history is miserable.

The metric catalogue for evaluating tool calls

Do not implement all seven this week. Start with tool-selection F1 and JSON-schema validity. Both are deterministic and effectively free. Add required-argument matching and order sensitivity once selection sits above roughly 0.9, because until then selection noise swamps everything downstream. Bring in an LLM judge last, and only for questions code cannot decide.

Cost drives the shape of the whole programme. Deterministic checks run in milliseconds against data you already have, so they can run on every sampled trace. Judges cost model tokens per evaluation and must be sampled.

Metric	What it catches	Ground truth needed?	Code or judge	Per-run cost
Tool-selection precision / recall / F1	Wrong or missing tool	Yes	Code	Negligible
JSON-schema validity	Type errors, hallucinated fields	No	Code	Negligible
Required-argument match	Wrong values in known fields	Yes	Code	Negligible
Semantic argument similarity	Free-text args (queries, bodies)	Yes	Embedding or narrow judge	Low to moderate
Order / trajectory score	Broken dependency order, loops	Partial (budget checks need none)	Code	Negligible
Unnecessary-call rate	Over-calling, wasted API spend	Yes (empty expected lists)	Code	Negligible
Recovery-after-error	Fabrication after a failed call	No	Judge, or code on retry patterns	Moderate

Tool selection accuracy, and why exact match lies

Definition: the fraction of steps with a ground-truth call where the emitted tool name matches. Simple. It also has two failure modes that will mislead you.

Multi-tool queries break it first. LangWatch’s worked example, “Check my calendar for next week’s meetings and set reminders for each one”, scores precision 1.0 and recall 0.5 because one of the two required tools fired (LangWatch cookbook). A naive exact-match scorer records that as a clean pass or a total fail, depending on whether the implementer compared sets or lists. Neither is right.

Format compliance is the second confound. T-Eval reports that ChatGLM3-6B and Baichuan2-7B score around 80% on its Instruct subset under a JSON protocol, while planning scores rise by 25 points when the same models are evaluated in string format (T-Eval). You can measure JSON-rendering skill and believe you measured tool choice. Log parse failures as their own score. Never fold them into tool-selection misses.

Assert on structured fields. OpenAI hands you a tool_calls array with name, arguments, id and index. Anthropic hands you tool_use blocks. Google’s Gemini API hands you function-call parts. Regexing raw text for a function name is how you end up scoring a sentence that merely mentions the tool.

Argument accuracy in three layers

Run them in order and short-circuit.

Layer 1: schema validity. Validate the arguments object against the tool’s JSON Schema. Binary, free, and it catches type coercion errors and invented field names. This gates everything downstream. An argument comparison on an object that never parsed is meaningless.

Layer 2: required-field match with an ignore-list. Score only the fields present in the expected output. An extra include_history=false on an otherwise perfect call is not a failure. Treating it as one teaches your dashboard to lie.

Layer 3: semantic match for free-text arguments. Search queries, email bodies, summarisation prompts. Exact match is hopeless here. Use embedding similarity, or a judge prompt scoped to that single field. T-Eval measures its “understand” dimension this way, comparing generated arguments against golden arguments by similarity rather than equality (T-Eval).

Most argument failures logged in practice are normalisation bugs in the scorer, not model errors. Work through this checklist before you believe a single argument miss:

• Case and surrounding whitespace on string values
• Date and datetime formats (2026-08-19 vs 19/08/2026 vs epoch seconds)
• ID prefixes (ord_12345 vs 12345)
• Units and currency minor units (£12.50 vs 1250)
• Numeric type ("5" vs 5)
• null vs key absent vs empty string

A Pydantic model or a shared JSON Schema file, used by both the tool definition and the scorer, removes most of this class. Generate one from the other. Do not hand-maintain two copies.

Precision, recall and F1 over the tool set

Precision equals correct calls divided by total calls made. Recall equals correct calls divided by calls that should have been made. F1 is their harmonic mean. State the formulas in your codebase. Published definitions in vendor cookbooks are loose enough that two teams reading the same page will implement different things.

Precision matters more here than in retrieval. This is the argument to internalise. In RAG, over-retrieval is nearly free: pull ten chunks, the model ignores the seven irrelevant ones. In tool calling there is no filter downstream. An unnecessary send_email actually sends the email. Every extra call costs latency, input tokens on the next turn, and a side effect you cannot retract (LangWatch cookbook).

Break recall down per tool. An aggregate F1 of 0.78 across five tools usually means four tools near 0.95 and one near 0.2, and the fix path depends entirely on which one:

Tool	Calls expected	Calls made correctly	Recall	Likely fix
get_calendar_events	12	12	1.00	None
create_reminder	9	6	0.67	Sharpen description; disambiguate from calendar tool
send_email	6	2	0.33	Rename function; check whether the model is avoiding side effects

Illustrative shape, using the per-tool recall figures published in LangWatch’s cookbook run.

F1 is the wrong shape for single-tool-per-turn agents. Build a confusion matrix over tool names instead. It shows which pairs the model conflates, which F1 averages away.

Order and trajectory, scoring a sequence rather than a set

Order is scoreable only where a data dependency exists. If the agent must resolve today’s date before it can query last month’s weather, order is part of correctness. T-Eval uses that exact example (T-Eval). If two independent lookups can happen in either order, penalising order is scoring noise.

Four trajectory checks need no ground truth and cost nothing:

1. Step count against a budget
2. Duplicate-call detection (same name, same normalised args, twice)
3. Repeated-failing-call detection (same call, same error, twice)
4. Required-step presence (did an authenticate call happen before any write?)

The ground-truth version is heavier. T-Eval matches predicted against golden action sequences using Sentence-BERT similarity plus Hopcroft-Karp maximal matching, then scores the longest ordered sub-sequence within that pairing (T-Eval). Why bipartite matching rather than an index-by-index comparison? If the agent inserts one extra step at position 2, an index-wise comparison misaligns everything after it and reports near-zero. Matching first, then scoring order within the matched pairs, tolerates one insertion or one skip and still gives you a graded score.

If you want ordering signal without the machinery, compute normalised Levenshtein distance over the sequence of tool names. Ten lines, no embeddings, and it degrades gracefully on insertions. Code below.

Leaping’s Launch HN post describes building voice agents as an explicit multi-stage graph, so that a regression traces to one stage rather than to “the call”. That is a founder’s account of their own product design, not an independent finding. The instrumentation lesson still stands. Your trajectory score is only as diagnostic as your span boundaries.

The metric nobody publishes: unnecessary-call rate

Benchmarks and vendor cookbooks share one assumption. The correct answer always contains a tool call. An agent that calls a function every turn therefore scores perfectly while burning API fees, adding a round-trip of latency, and occasionally sending an email nobody asked for. When2Tool is built to attack that assumption.

The metric:

unnecessary_call_rate = calls made on tasks answerable without tools
                        ------------------------------------------
                                    total calls made

Report it beside accuracy, always as a pair. One number alone tells you nothing about the trade-off you just made.

Prompt-level control of this behaviour is coarse. When2Tool reports that switching from a default prompt to a “use tools sparingly” prompt costs Qwen3-14B 27.3 accuracy points on hard tasks to save 0.47 tool calls, and that a Reason-then-Act prompt on Llama-3.3-70B costs 63.3 accuracy points on hard tasks (When2Tool). Sparse prompting suppresses calls indiscriminately, and the tasks that most need a tool lose the most.

The paper’s own method points at a better ceiling. A linear probe on pre-generation hidden states predicts tool necessity at AUROC 0.89 to 0.96 across six models, from Qwen3-1.7B at 0.894 through Qwen3-14B at 0.957. Their Probe&Prefill method cuts calls by 48% for 1.7% accuracy loss, against baselines that cut 6% at comparable accuracy or lose five times the accuracy. It generalises to agentic search, reducing API calls 20 to 56% on Search-o1 (When2Tool).

Be clear about what that means for you. It is a single-lab preprint, it requires access to hidden states, and no team building on a closed API endpoint can implement it.

What you can do this week: carve a no-tool-needed slice into your eval set, 10 to 20% of items whose expected_tool_calls is an empty list, and score it as its own number. That single change surfaces over-calling regressions every other metric on this page is blind to.

Replotted from the published ΔAcc and ΔTC figures in When2Tool Tables 1 and 3, model names and difficulty tiers intact. Reproduced from that source, not measured here.

Error recovery, and the operational scores to log anyway

After a tool returns an error, the agent has four options. Retry with corrected arguments, switch tools, hand back to the user, or fabricate a result and continue. Only the last is unambiguously wrong, and it is the one output-only evaluation never catches. T-Eval’s five review labels give you a ready-made rubric (T-Eval).

Recovery hides because it usually works. The agent retries, gets its data, produces a correct final answer, and your output eval scores 1.0 on a run that burned two extra model calls and four seconds of user-visible latency.

Log per-call latency, token cost and tool error rate as numeric scores alongside quality scores. Then define one composite for alerting: the percentage of traces that were correct and under step budget and had zero tool errors. That single line on a dashboard catches degradations no individual metric flags.

Parallel, streaming and MCP calls change what you score

Parallel tool calls arrive as several entries in one assistant message. The index field tells you their position in that batch, and it is not a sequence. Scoring a parallel batch with an order metric manufactures failures. Detect the batch first, score it as a set, and reserve the order score for calls that span turns.

Streaming complicates the capture, not the metric. Argument JSON arrives in deltas and is only valid once the stream closes. Accumulate, then validate. A scorer that validates mid-stream reports schema errors that never existed.

Model Context Protocol servers add a third wrinkle. The tool list is dynamic, so the set of available_tools the model saw at run time may differ from the set it sees today. Record the tool list on the trace. Without it, a recall drop is unattributable: you cannot tell whether the model got worse or the server stopped advertising the function.

Building the ground-truth set, with a copy-pasteable template

Keep the format flat and vendor-neutral so it survives a platform migration. CSV for humans, JSONL for the harness, same columns:

Column	Type	Notes
id	string	Stable; never reuse after deletion
user_input	string	The turn under test
message_history	JSON array	Prior assistant and tool messages. The column everyone forgets
available_tools	JSON array	Schemas the model saw, not the ones it should have seen
expected_tool_calls	JSON array of {name, args, required_args_only}	Empty array for the no-tool slice
expected_no_tool	bool	Explicit, so an empty array is never ambiguous with a missing value
acceptable_alternatives	JSON array of arrays	For ambiguous cases where two tools are both defensible
must_not_call	JSON array of names	Hard-fail semantics in CI
difficulty	enum	easy / medium / hard, for slice-level reporting
source_trace_id	string	Links the case back to the production failure that created it
notes	string	Why this case exists

Source items from a loop, never from imagination. Production trace with negative feedback, then labelling queue, then a human writes the expected output, then a dataset item, then a permanent regression case. Both Laminar and Langfuse document this loop in their own products.

On size, two things are true at once. LangWatch’s cookbook argues that three or four tools and a handful of cases are enough to stand the harness up, which is right for getting to first signal (LangWatch cookbook). A five-row set also swings 20 percentage points when a single item flips, so it cannot gate a merge. Fifty items is a floor for a gate. A hundred is comfortable.

Compose deliberately. Single-tool cases, multi-tool cases, ordered-dependency cases, no-tool-needed cases, must-not-call cases. Encode “either tool is acceptable” in acceptable_alternatives as a list of complete acceptable call lists, and pass if the actual calls match any one of them. Scoring against a single canonical list on a genuinely ambiguous query manufactures failures.

The message_history trap deserves its own warning. Evaluating tool calls nearly always needs the prior assistant and tool messages, and dataset tooling drops them. A Langfuse user reported in Discussion #9554 (October 2025) that adding a traced interaction to a dataset through the UI captured only the message and metadata, with tool calls absent, and was directed to iterate the dataset through the SDK instead. Check whether that has since shipped before you rely on any UI import path.

Five example rows over a generic calendar, email and reminder tool set, including one no-tool-needed row and one must-not-call row. CSV and JSONL published as downloadable files.

Code evaluator, LLM judge, or human

One rule decides most cases. If the check is decidable from the recorded tool_calls array plus the tool schema, it is code. If it requires reading intent, it is a judge. If it requires domain expertise, or it is the calibration set the judge is measured against, it is human.

Code handles: required tool called, arguments validate against schema, no duplicate calls, step budget respected, no-tool cases stayed tool-free, no must_not_call tool appeared.

A judge handles: was this tool appropriate for this request, was the free-text argument a reasonable rendering of intent, did the agent handle an error response sensibly.

Scope judge prompts to one question each. Asking a judge to rate “overall tool use quality” out of five produces a number that correlates with nothing. Pass only the data the question needs. For a selection question, pass just the tool names; a JSONPath-style selector like $[*].name is the pattern Langfuse documents. For an argument-quality question, pass the full argument array.

Humans do the work neither can. Annotation queues exist to produce the reference set a judge is calibrated against. Without periodic recalibration, a judge’s drift is invisible to you.

State this plainly to your team. Measure agreement between your judge and human raters on a sample of your own domain before any judge score gates a merge. No published figure transfers, because agreement depends on your tools, your rubric and your judge model. This page cannot give you that number and neither can a vendor page.

Decidable from the tool_calls array plus the schema? Code. Requires reading intent? Judge. Is it the calibration set? Human. All seven metrics from this page appear as leaves.

A minimal Python harness you can run today

Framework-free scorer first. All code here is illustrative. Verify library versions against current documentation before pinning anything.

import json
from difflib import SequenceMatcher
from jsonschema import Draft202012Validator, ValidationError

def norm_name(n: str) -> str:
    return (n or "").strip().lower()

def norm_value(v):
    if isinstance(v, str):
        return " ".join(v.strip().lower().split())
    if isinstance(v, list):
        return [norm_value(x) for x in v]
    if isinstance(v, dict):
        return {k: norm_value(x) for k, x in sorted(v.items())}
    return v

def schema_valid(call, tool_schemas) -> bool:
    schema = tool_schemas.get(call["name"])
    if schema is None:
        return False                      # hallucinated tool name
    try:
        Draft202012Validator(schema).validate(call["args"])
        return True
    except ValidationError:
        return False

def required_args_match(expected_args, actual_args) -> bool:
    """Compare only fields present in the expected call. Optional extras allowed."""
    for k, v in expected_args.items():
        if k not in actual_args:
            return False
        if norm_value(v) != norm_value(actual_args[k]):
            return False
    return True

def order_score(expected, actual) -> float:
    """Normalised sequence similarity over tool names. Levenshtein-style fallback
    for teams who want ordering signal without embeddings or bipartite matching."""
    e = [norm_name(c["name"]) for c in expected]
    a = [norm_name(c["name"]) for c in actual]
    if not e and not a:
        return 1.0
    return SequenceMatcher(None, e, a).ratio()

def score_case(expected, actual, tool_schemas, expects_no_tool=False):
    exp_names = [norm_name(c["name"]) for c in expected]
    act_names = [norm_name(c["name"]) for c in actual]

    matched = 0
    pool = list(exp_names)
    for n in act_names:
        if n in pool:
            pool.remove(n)
            matched += 1

    precision = matched / len(act_names) if act_names else (1.0 if not exp_names else 0.0)
    recall    = matched / len(exp_names) if exp_names else (1.0 if not act_names else 0.0)
    f1 = 0.0 if precision + recall == 0 else 2 * precision * recall / (precision + recall)

    all_valid = all(schema_valid(c, tool_schemas) for c in actual) if actual else True

    arg_hits = 0
    for e in expected:
        for a in actual:
            if norm_name(a["name"]) == norm_name(e["name"]) \
               and required_args_match(e["args"], a["args"]):
                arg_hits += 1
                break

    return {
        "tool_name_precision": round(precision, 3),
        "tool_name_recall": round(recall, 3),
        "tool_name_f1": round(f1, 3),
        "schema_valid": all_valid,
        "required_args_match": round(arg_hits / len(expected), 3) if expected else 1.0,
        "order_score": round(order_score(expected, actual), 3),
        "unnecessary_calls": len(actual) if expects_no_tool else max(0, len(act_names) - matched),
        "parse_failures": sum(1 for c in actual if not c.get("parse_ok", True)),
    }

Wiring it to a provider matters more than it looks. Read the structured field. Do not parse content.

def extract_openai(response) -> list[dict]:
    msg = response.choices[0].message
    out = []
    for c in (getattr(msg, "tool_calls", None) or []):   # [] when the model answered directly
        try:
            args, ok = json.loads(c.function.arguments), True
        except json.JSONDecodeError:
            args, ok = {}, False
        out.append({"id": c.id, "name": c.function.name, "args": args, "parse_ok": ok})
    return out

def extract_anthropic(message) -> list[dict]:
    return [
        {"id": b.id, "name": b.name, "args": b.input, "parse_ok": True}
        for b in message.content if getattr(b, "type", None) == "tool_use"
    ]

That or [] is the whole no-tool-needed slice. Without it, every empty-expected case raises before it is ever scored.

Platform-integrated evaluators take the same logic and read the tool-call field off an observation, returning several named scores from one run: a boolean used_required_tool, a numeric tool_call_count, a boolean schema_valid. Langfuse documents the contract as ctx.observation.tool_calls, or toolCalls in JS, with id, name, arguments, type and index per entry, and describes these evaluators as running without network egress (Langfuse guide). The equivalent elsewhere is the dataset-plus-evaluator pattern Laminar documents. Verify both against current docs before writing against them.

Expect two outputs. A per-case table with query, expected, actual, precision, recall and latency. A per-tool rollup, which is the shape LangWatch’s cookbook results take.

One synthetic three-tool query with fixed expected and actual call arrays, scored by exact match, name-only F1, required-args-only match, order-aware score and unnecessary-call penalty. The same run reads anywhere from 0.0 to 1.0 depending on metric choice. Every figure hand-computed from the formulas above.

Where evaluating tool calls fits your existing stack

You do not need a new framework to start. You need the field. LangChain and LangGraph, LlamaIndex, CrewAI, AutoGen, the OpenAI Agents SDK and Pydantic AI all surface tool calls as structured records on their run or event objects. Find that record, map it to the {name, args, parse_ok} shape above, and every scorer on this page works unchanged.

The same applies to eval libraries. Ragas, DeepEval, Arize Phoenix, Braintrust, Comet Opik, TruLens and Weights & Biases Weave each accept a custom scoring function. The seven metrics here are custom scoring functions. Hosted platforms such as Vertex AI, Azure AI Foundry and Amazon Bedrock expose their own evaluation surfaces, and the same rule holds: read their documented tool-call field before writing a scorer against it.

Adapter, not rewrite. Keep the scorers in plain Python, keep the dataset in CSV and JSONL, and treat the platform as transport. Then a migration costs you one function.

Offline experiments versus online evaluation

	Offline experiment	Online evaluation
When it runs	Every PR, every model swap	Continuously, on sampled live traces
Data	Pinned dataset version	Real user traffic, no expected output
Catches	Regressions against known failures	Drift, novel inputs, new failure classes
Metrics available	All, including ground-truth comparisons	Reference-free only, plus judges
Cost driver	Dataset size × runs per PR	Sampling rate × judge token cost
Blind spot	Anything not yet in the set	Cannot prove a fix, only detect a symptom

Offline sets only contain failures you already know about. That is not a criticism. It is the definition. Online evaluation is the only mechanism by which a new failure class enters your dataset, which makes the pipeline from production trace to dataset item the most load-bearing part of the system.

Design sampling from a budget, not from a round number. Run the free code checks on every sampled trace. Then pick the judge sampling rate by dividing a monthly spend ceiling by the per-evaluation cost:

evals/month = monthly_judge_budget / (tokens_per_eval × price_per_token)
sampling_rate = evals_per_month / traces_per_month

A tool-call judge evaluation is usually a short rubric plus the call array, so the prompt is small compared with a RAG faithfulness judge. Substitute your provider’s current published pricing for your judge-tier model, and your own measured token count per evaluation.

Filter evaluators to the right observations, or you will score the wrong node. Mark tool spans and retriever spans with distinct types at instrumentation time. Laminar documents span_type="TOOL" (Laminar guide) and Langfuse documents a retriever observation type for the same purpose (Langfuse guide). Both are OpenTelemetry-shaped, so the attribute travels if you migrate.

Tag traces when a user gives negative feedback, then query the tagged population later to build dataset items. Laminar documents asynchronous tagging by trace ID for exactly this (Laminar guide).

The pipeline that turns a live failure into a permanent regression case, with the free code checks running on every sampled trace and the judge sampled against a budget.

Gating agent changes in CI

Four categories of change need a gate. Prompt edits, model swaps, tool-schema edits, orchestration changes. There is a fifth that teams routinely miss.

Tool description edits. A one-word change to a docstring changes selection behaviour, and no type checker, no linter and no unit test will notice.

The framework-agnostic gate shape: run the harness against a pinned dataset version, compute per-metric aggregates, fail the job on a threshold miss, post the per-case diff as a PR comment. The reviewer then sees which cases flipped rather than a single red number.

Gate on deterministic metrics only. Schema validity, required-tool-called, step budget, must-not-call violations. Those are hard thresholds. Report judge scores in the PR comment and do not gate on them until you have measured judge and human agreement on your own domain.

Two productised versions, both vendor documentation. Langfuse ships a CI gate as langfuse/experiment-action@v1.0.0, requiring Python SDK v4.6.0+ or JS SDK v5.3.0+, with dataset_version pinning for reproducibility, and marks its evaluator-management API endpoints unstable (Langfuse guide). Laminar’s changelog documents lmnr-cli, which lets a Signal’s prompt and output schema live in the repository and ship from CI, with trigger, filter and mode as three independent flags (Laminar changelog). Re-check both version numbers before you copy them into a workflow file.

Handle nondeterminism explicitly. Pin temperature to zero where the provider honours it. For anything still stochastic, run N samples per item and gate on the mean with a variance guard, so one unlucky run does not block a merge and a genuinely noisier model does not sneak through.

Pinned dataset, deterministic thresholds hard-fail, judge scores reported without gating, per-case diff posted to the PR.

Platform capability matrix

Methodology note: every cell below comes from public documentation, changelogs and issue trackers. Nothing here was installed, configured or measured. Claims that appear only on a vendor’s own pages are labelled vendor claim. “Unknown” means the public pages read for this table did not settle the question, and it is an invitation to check, not a negative finding. Limitations come from public issue trackers, never from a competitor’s comparison page. Re-verify each row on the day you decide.

Platform	Structured tool-call field for evaluators	Code evaluators	Observation-level LLM judge	Session / multi-turn scores	Dataset + labelling queue	CI action or CLI	Self-host	Licence
Langfuse	Yes, tool_calls / toolCalls with id, name, arguments, type, index (vendor claim)	Yes, documented as running without network egress (vendor claim)	Yes, and trace-level judges described as legacy (vendor claim)	Yes, session-level scores documented	Yes	langfuse/experiment-action@v1.0.0, Python SDK v4.6.0+ / JS v5.3.0+ (vendor claim)	Yes	Unknown
LangWatch	Unknown	Yes, cookbook publishes a tool-selection scorer	Unknown	Unknown	Unknown	Unknown	Yes; 3.0 rebuilt self-hosting on ClickHouse, Helm or Docker Compose, no Elasticsearch (vendor changelog)	Unknown
Laminar	Span typing via span_type="TOOL" (vendor docs)	Yes, dataset plus evaluator pattern	Unknown	Unknown	Yes, plus async trace tagging	lmnr-cli Signals, prompt and schema in-repo (vendor changelog)	Unknown	Unknown
Maxim AI	Tool-call accuracy evaluator described in vendor guide	Vendor claim	Vendor claim	Unknown	Unknown	Unknown	Unknown	Commercial
Ragas (OSS)	Library-level; you pass structures in	Yes, Python	Yes	Unknown	Bring your own	Any test runner	N/A (library)	Unknown
OpenTelemetry + your own scorer	Whatever you instrument	Yes	Bring your own	Yes, via span attributes	Bring your own	Any test runner	Yes	Apache-2.0

Known limitations, from public trackers only. Langfuse Discussion #9554 (October 2025) reports that adding a traced interaction to a dataset via the UI captured only the message and metadata, with tool calls absent. The response directed the user to iterate datasets through the SDK. Check whether it has since shipped. Maxim’s public guide imports from maxim_py in one example and maxim elsewhere, an inconsistency worth resolving against current SDK docs before writing code against it. For LangWatch, Laminar and the two open-source rows, pull current issues from the langwatch/langwatch, lmnr-ai/lmnr and relevant upstream trackers rather than trusting an aged table.

How to verify this yourself. Read three pages per row. The evaluator or scoring API reference. The changelog for the last ninety days. The open-issues list filtered to “eval” or “tool”. Record the URL and the doc date per cell. That takes about twenty minutes per platform, and it is the only way this table stays true.

What T-Eval, When2Tool, BFCL and friends actually measure

Benchmark	Decomposition	Size	Ground truth	Headline metric	Live API calls?	What it does not measure
T-Eval	plan, reason, retrieve, understand, instruct, review	Instruct 2,660; Retrieve 6,426; Plan 553; Reason 6,426; Review 487	Human-annotated golden paths	Per-ability scores; planning via Sentence-BERT + Hopcroft-Karp, longest ordered sub-sequence	No, deliberately avoided	Whether a tool was needed at all
When2Tool	tool necessity across three difficulty tiers	18 environments (15 single-hop, 3 multi-hop); 1,080 train, 2,700 test	Task construction with known tool necessity	Accuracy paired with tool-call count (ΔAcc / ΔTC)	Unknown	Argument quality; multi-step ordering depth
BFCL	Function-calling categories; read the current Berkeley leaderboard for names and sizes	Unknown	AST matching plus executable checks, per the leaderboard	Accuracy per category	Partly, executable subset	Unknown
ToolBench	real API tool use with retrieval over a large API pool	Unknown	Unknown	Pass rate and win rate	Yes	Unknown
API-Bank	Read the paper for the current ability split	Unknown	Unknown	Unknown	Unknown	Unknown
Gorilla	API selection over ML model hubs	Unknown	Unknown	AST accuracy and hallucination rate	Unknown	Unknown
tau-bench	conversational tasks with a user simulator	Unknown	Database end-state comparison	pass^k reliability	Unknown	Unknown

Only the T-Eval and When2Tool rows carry figures sourced here (T-Eval; When2Tool). Fill the rest from each project’s primary paper or official leaderboard before publishing, and never carry over one paper’s description of another. That ages badly.

T-Eval’s design choice deserves borrowing regardless of its scores. It avoids live API calls, so API instability and temporal drift do not pollute results.

Treat every model ranking on these boards as historical. T-Eval’s experiments ran 12/01/2023 to 12/10/2023, and its figures describe GPT-3.5, Claude2, LLaMA2 and Qwen-72B. Its headline observations, GPT-4 reaching 95% on the review dimension while most models sat at 50 to 60%, and Qwen-72B trailing GPT-3.5 by more than 20 points, are a snapshot of models two or more generations old (T-Eval).

The transferable asset is the metric decomposition, not the ranking. A benchmark score tells you nothing about how a model handles your twelve tools with your descriptions. Build the domain eval set.

Evaluating tool calls for safety, not just correctness

An agent with MCP access can query production databases, push commits, post to Slack and run shell commands. “Did it pick the right tool” and “should it have been permitted to run that tool with those arguments” are different questions. Only the first has a metric in most published guides.

Policy evaluation is a pre-execution gate. That makes it structurally different from post-hoc scoring. An open-source project posted to Hacker News describes a policy engine sitting between agents and MCP servers, evaluating every tool call against YAML security policies before execution, on the stated premise that MCP-connected agents have this reach with no security layer in between. That is the author’s description of their own project, not a market survey.

Checks worth adding to the eval set:

• Destructive-tool calls that must be preceded by a confirmation step
• Argument-level scoping (does the delete_records call carry a WHERE-equivalent filter, or is it unbounded?)
• Blast-radius scoring: rank tools by irreversibility and weight precision failures by that rank
• Credential and scope checks (did the agent reach for an admin-scoped tool on a user-scoped request?)

Here the precision argument stops being about efficiency. For side-effecting tools, a precision failure is an incident. The over-called send_email actually sent.

Give must_not_call hard-fail semantics in CI. One violation fails the job regardless of aggregate scores. Averaging a security failure into an F1 is how it ships.

Tools ranked by irreversibility, with the policy gate sitting between the agent and the MCP server, and must-not-call violations hard-failing the CI job.

System-level failures your harness will blame on the model

An operator running a benchmark-style evaluation of a production agent reported on Hacker News that most failures were system-level rather than model quality. Broken URLs inside tool calls dropped a score to 22, and the agent called localhost from a cloud environment. That is one practitioner’s account of one system, and it is worth more than a generalisation because you can go read it.

Build the catalogue outward from there:

• Environment mismatch. Staging or localhost endpoints baked into tool definitions that were written on a laptop.
• Auth and credential expiry. A rotated key surfaces as a tool error, and an output-only eval reads it as a reasoning failure.
• Rate limits and timeouts. Especially on parallel calls, where the agent’s correct behaviour triggers the throttle.
• Tool-schema drift. The schema the model sees diverges from the function actually invoked, so a perfectly-formed call fails at execution.
• Nondeterministic tool responses. A live search API returns different results every run, and your expected outputs rot.

The diagnostic rule is one line of instrumentation. Split tool-call failures into well-formed call, execution failed versus malformed call. The first is an infrastructure bug. The second is a model or prompt bug. Most teams never make this split, and spend a week rewriting a prompt to fix an expired token.

Freeze tool responses in CI and run live tools only in staging. T-Eval’s stated reason for avoiding live API calls is exactly this: external instability distorts scores (T-Eval).

Watch for cascading failures. One bad early call poisons every downstream step, so a trace-level score reports five failures where there was one cause. Per-step scoring makes the root cause visible, a pattern also described in Maxim’s guide.

Ten mistakes that make tool-call evals useless

1. Exact JSON string equality on arguments. An extra optional field fails a correct call. Compare required fields only, with normalisation.
2. Measuring format compliance and calling it tool choice. T-Eval’s 25-point string-versus-JSON gap is the evidence (T-Eval). Score parse failures separately.
3. No no-tool-needed cases. Your over-calling rate is unmeasured and therefore unbounded.
4. Set comparison on order-dependent tasks. A retry loop scores like a clean run. Add an order score where a data dependency exists.
5. Parse failures counted as wrong-tool. Two bugs, two fixes, one meaningless number.
6. Judge scores in a merge gate before agreement is measured. Gate on deterministic checks. Report the judge.
7. An eval set that never grows from production. Offline sets only contain yesterday’s failures unless online evaluation feeds them.
8. Message history dropped when building dataset items. Reproduce Discussion #9554 and you will score a turn the model never saw in context.
9. One aggregate score instead of per-dimension scores. Five failure classes, one number, zero diagnostic value.
10. Live tools in CI. Reruns disagree, the gate flakes, and someone disables it.

Frequently Asked Questions

How do you evaluate tool calls in an LLM agent?

Score five things separately. The decision to call a tool at all, the tool name selected, the arguments passed, the position of the call in the sequence, and how the agent handled the response including errors. Start with tool-selection F1 and JSON-schema validity as deterministic code checks. Both run on the recorded tool_calls array, not on parsed model text, and both cost effectively nothing per trace. That means you can run them on every sampled production trace as well as in CI.

What metrics measure tool call accuracy?

Seven. Tool-selection precision, recall and F1 over names. Argument schema validity. Required-argument match with optional extras ignored. Order or trajectory score. Unnecessary-call rate on tasks answerable without tools. Tool error rate. Recovery-after-error. A single aggregate “tool accuracy” number cannot tell you which of the five failure classes you have. Compute the split from the start. Retrofitting it after months of dashboard history means throwing that history away.

What is a good tool call eval set, and how big does it need to be?

A good set carries eleven columns, including message history, available tools, expected tool calls, an explicit no-tool flag, acceptable alternatives for ambiguous cases and a must-not-call list. Every item traces back to a real production failure by trace ID. Composition matters more than size: single-tool, multi-tool, ordered-dependency, no-tool-needed and must-not-call slices. Fifty items is the floor for a merge gate. A five-row set swings twenty points when one item flips.

How do you evaluate whether an agent should have called a tool at all?

Add cases whose expected tool list is empty, score them as a separate slice, and compute unnecessary-call rate as calls made on tool-free-answerable tasks divided by total calls. Report it alongside accuracy so you see the trade-off rather than one number. Prompting is a blunt instrument here. When2Tool reports a sparse prompt costing Qwen3-14B 27.3 accuracy points on hard tasks to save 0.47 calls (When2Tool).

Should evaluating tool calls use code or LLM-as-a-judge?

Use code for anything decidable from the tool_calls array plus the tool schema. Required tool called, schema validity, duplicate detection, step budget, must-not-call violations. Those are free and run in milliseconds on every sampled trace. Use a judge only for semantic questions such as tool appropriateness or free-text argument quality, at model cost per evaluation. Do not let judge scores gate a merge until you have measured judge and human agreement on a sample of your own domain.

How do you evaluate multi-step tool calls where order matters?

Order is scoreable only where a genuine data dependency exists, such as resolving today’s date before querying last month’s weather. Penalising order between independent lookups scores noise. Two implementations work. Deterministic checks needing no ground truth, meaning step budget, duplicate calls, repeated failing calls and required-step presence. And sequence matching against a golden path. T-Eval uses Sentence-BERT similarity plus Hopcroft-Karp matching, then scores the longest ordered sub-sequence (T-Eval).

Is there a template for tracking tool call evaluations in a spreadsheet?

Yes. Keep a flat CSV or JSONL with id, user_input, message_history, available_tools, expected_tool_calls, expected_no_tool, acceptable_alternatives, must_not_call, difficulty, source_trace_id and notes. Then add two derived tabs: per-case scores and a per-tool recall rollup. Message history is the column people forget, and it is the one that breaks evaluating tool calls. Some dataset UIs drop tool calls entirely when importing a trace (Langfuse Discussion #9554, October 2025).

How do you evaluate tool calls without ground truth?

Three reference-free options work on live traffic where no expected output exists. Property checks that need no reference, meaning schema validity, step budget, no duplicate calls and no must-not-call tools. An LLM judge with a rubric scoped to tool appropriateness. Pairwise comparison of two agent versions on the same input. This is how online evaluation catches drift and novel inputs. These detect a symptom. Only an offline set with expected outputs proves a fix.

Why do my tool call evals fail when the model looks fine?

Because the failure is usually below the model. Split every failing call into “well-formed call, execution failed” versus “malformed call”. The first is infrastructure. The second is model or prompt. One operator reported that most failures in their production agent evaluation were system-level, with broken URLs in tool calls dropping a score to 22 and the agent calling localhost from a cloud environment (Hacker News). Add auth expiry, rate limits and schema drift.

Should tool call evaluations run in CI?

Yes. Gate prompt edits, model swaps, tool-schema changes and orchestration changes, plus the one teams miss: tool description edits, which change selection behaviour with nothing in the type system to catch them. Run against a pinned dataset version, hard-fail on deterministic thresholds, report judge scores without gating on them, and pin temperature to zero or average N samples with a variance guard.

What is the difference between tool call evaluation and agent trajectory evaluation?

Tool-call evaluation scores individual calls. Was this the right function, do these arguments validate, do the values match the expected ones. Trajectory evaluation scores the path: how many steps, in what order, with what retries and loops, at what token cost and latency. A run can pass every individual tool-call check and still fail the trajectory budget by taking eleven steps to do a three-step job.

Where to start on Monday

Two scorers. Tool-selection F1 and JSON-schema validity, both reading the structured tool_calls field, both running on every sampled trace for free. Then fifty dataset rows sourced from real failures, with a no-tool-needed slice carved out and a must_not_call column that hard-fails the build. Everything else on this page is an upgrade path from there. Evaluating tool calls well is mostly a matter of refusing to collapse five independent decisions into one number, and of linking every figure you publish back to the source it came from.

---

Related reading: LLM-as-a-judge calibration · Agent trajectory evaluation · Eval datasets from production traces · LLM observability tools compared · MCP security policy gates · Tool descriptions and selection accuracy · Gating LLM changes in CI · Function calling schema design · Session-level agent evaluation · RAG evaluation metrics