Definition
The tool selection accuracy test checks whether your agent calls the right tools: every tool a request needs, and nothing else. Each row is one agent run, with its trace and a column that lists the tools the run should use. The test compares the tools the trace called with that list and reports tool precision, recall, and F1. You can also list distractor tools, which look relevant but are the wrong choice, and track how often your agent picks them.Taxonomy
- Task types: LLM.
- Availability: and .
Why it matters
- An agent that skips a tool answers without the data it needs, and an agent that calls extra tools adds latency, cost, and risk. Recall catches the first problem and precision the second.
- Distractor tools surface a specific routing mistake: reaching for a tool that looks right for the request but is not.
- The test checks the agent’s decisions directly, so you catch a tool-routing regression even when the final answer still looks plausible.
Required columns
To compute this test, each row needs:- Trace: The agent run’s trace. The test reads the steps of type
tooland names each tool by its function name, or by the step name when there is no function name. In monitoring, instrument your code to send traces. In development, the Openlayer SDK records each row’s trace when therunmethod of your run script calls traced code. - Expected tools: A column that lists the tools the row should use, as a
JSON list (
["search", "lookup"]) or comma-separated names (search, lookup). Use[]when the row should use no tool.
How it works
For every row, the test compares the set of tools the trace called with the set the row expects. Tool names match regardless of case and surrounding spaces, and each tool counts once per row, however many times the agent called it.- Precision is the share of the tools used that were expected.
- Recall is the share of the expected tools that were used.
- F1 is the harmonic mean of precision and recall.
- Distractor tool rate is the share of the tools used that are distractor tools.
macro averaging, those
rows are left out of that measurement’s average. Their F1 is 0.
The Averaging setting decides how rows combine:
macro(default) scores each row and then averages the rows, so every row counts equally.micropools the tool matches from all rows before computing the scores, so rows that use or expect more tools count more.
Example
Take five rows, withweather as a distractor tool:
With
macro averaging, tool precision is (1 + 1/3 + 1 + 0) / 4 ≈ 0.583, tool
recall is (1 + 1/2 + 1 + 0) / 4 = 0.625, and tool F1 is 2.4 / 5 = 0.48. The
distractor tool rate averages rows 1, 2, and 4, the rows that used a tool:
(0 + 1/3 + 0) / 3 ≈ 0.111.
With micro averaging, the rows pool 3 correct tools, 3 unexpected tools, and
2 missed tools. Tool precision is 3 / 6 = 0.5, tool recall is 3 / 5 = 0.6, tool
F1 is 6 / 11 ≈ 0.545, and the distractor tool rate is 1 / 6 ≈ 0.167.
The test is skipped, not failed, when the expected tools column is missing,
when no row has trace data, when no row has a valid expected tools value, or
when no row called a tool and no row expects one. It is also skipped when the
selected measurement is undefined for the rows evaluated, such as the
distractor tool rate when no row used a tool. The skip message says which.
Parameters
Measurements
Thresholds are set on one of the measurements below. Every value is between 0 and 1.Test configuration examples
If you are writing atests.json, here are a few valid configurations for the tool selection accuracy test:
Related
- Unauthorized tool calls test - Check that an agent only calls tools from an allowed list.
- Instrument your code - Send traces from your agent to Openlayer.