How to Measure Tool-Calling Accuracy in AI Agents (2026)

An agent that calls the wrong tool, or the right tool with a broken argument, fails silently - it returns a confident answer built on a call that never should have run. You cannot catch that with an output-quality score alone, because the final text can look fine while the tool use underneath was wrong. Tool-calling accuracy is the metric that opens up the black box, and the trick to measuring it well is realizing it is not one number but three separate questions.

This guide decomposes those questions, shows how to score each one, and stays deterministic wherever the task allows - because the best thing about tool-call evaluation is how little of it needs an LLM judge.

The three questions inside “tool-calling accuracy”

When people say an agent “used its tools correctly,” they are collapsing three independent checks:

• Tool selection - did the agent pick the right tool for this step? Calling search_web when it should have called query_database is a selection error.
• Argument correctness - did it fill the parameters with the right values, in the right types? Right tool, wrong user_id, is an argument error.
• Call order - for multi-step tasks, did it call tools in a valid sequence? Booking a flight before checking availability is an ordering error even if both calls individually succeed.

Measure these separately, because each failure has a different cause and a different fix. A selection error usually points at the tool descriptions or system prompt. An argument error points at how the model parses the user request. An ordering error points at the agent’s planning loop. One rolled-up “62% accurate” number tells you something is wrong but not what - the decomposition tells you where to look.

Step 1: Capture the actual tool calls

You cannot score calls you did not record. Instrument the agent so every tool invocation lands in a trace as its own span with the tool name and the parsed arguments. Langfuse is the open-source default here - MIT-licensed, free to self-host, and it records nested spans so a multi-tool run shows up as a readable tree rather than a wall of logs. The point of capturing this is not just debugging one run - it is that your production traces become the raw material for the labeled dataset in Step 2.

Step 2: Build a labeled dataset with expected calls

For each test case, write down not just the expected final answer but the expected tool calls - which tool, which arguments, in which order. This is the ground truth you score against. Build it from real production traces and known failure cases rather than invented happy paths, because the messy edge requests are exactly where tool selection breaks down. Store it as a versioned eval dataset so a change to the agent is measured against a stable target.

A tool-call dataset needs one thing a plain QA dataset does not - the expected trajectory, not only the expected output. That is the whole reason this metric can catch what output scoring misses.

Step 3: Score each component - deterministically where you can

Here is the illustrative shape of a per-call scorer. Signatures vary by framework, so treat this as pseudocode:

# illustrative - not a specific SDK
def score_tool_call(expected, actual):
    selection = actual.name == expected.name
    # exact match for IDs/enums, tolerance for free-text
    args_ok = compare_args(expected.args, actual.args)
    return {"selection": selection, "arguments": args_ok}

def score_order(expected_seq, actual_seq):
    # were expected tools called in a valid order?
    return actual_seq == expected_seq   # or a subsequence check

Selection and order are exact-match comparisons - cheap, instant, and they never drift. Arguments are a blend - exact match for IDs, enums and numbers; tolerance or a semantic check only for free-text fields like a generated search query, where two different strings can both be correct.

You rarely build this harness from scratch. DeepEval ships a tool-correctness and task-completion metric among its 50-plus research-backed metrics and runs like pytest, so you write cases and run deepeval test run. Braintrust lets you drop scorers in as plain functions through its autoevals library, then compares runs in its experiments view. Both let you keep the deterministic checks deterministic instead of forcing everything through a judge.

Reach for an LLM judge only on the genuinely fuzzy slice - was a free-text argument reasonable - and never on tool name or call order, which a simple equality check grades perfectly for free.

Step 4: Run it several times and watch the distribution

Agents are non-deterministic, so a single pass is not a result. Run each case several times and look at how often tool selection holds, not whether it held once. A tool that is selected correctly four runs out of five is a real reliability bug hiding behind a green checkmark on the lucky run. This is the same discipline behind how to evaluate AI agents generally - measure the distribution, not one sample.

If you want to stress the agent on conversation paths you have not seen in production yet, Maxim simulates multi-turn users across many scenarios and personas, which surfaces tool-selection failures on inputs your captured traces never covered. Its honest trade-off is the bill - it meters per seat and caps logs at the same time.

Step 5: Wire it into CI as a regression gate

Once the metric is trustworthy, it belongs in the pipeline. Gate merges on it the same way you gate on unit tests - if a prompt change drops tool-selection accuracy on your dataset, the merge fails. Braintrust’s CI/CD quality gates can block a merge on a statistically significant regression, which is what turns a one-off measurement into ongoing regression testing. For the broader mechanics of that pipeline, see how to run LLM regression tests.

Common mistakes

• Scoring only the final answer. The output can be right by luck while the tool use was wrong - measure the calls, not just the text.
• Rolling everything into one number. Keep selection, arguments and order separate so the metric tells you where the bug is.
• Judging what you could assert. Tool name and order are exact-match checks - do not pay for an LLM call to compare two strings.
• Testing one run. Non-determinism means you need the distribution across repeats, not a single pass.

The bottom line

Tool-calling accuracy is three deterministic questions in a trench coat - right tool, right arguments, right order - and treating it as one fuzzy score is why teams miss silent tool-use bugs. Capture the calls in Langfuse, score each component with DeepEval or Braintrust, reserve the judge for free-text arguments only, measure across repeats, and gate CI on the result. For where this fits in the wider agent picture, read AI agent testing and how to evaluate CrewAI agents.