Skip to main content

Definition

The tool selection accuracy test checks whether your agent calls the right tools: every tool a request needs, and nothing else. Each row is one agent run, with its trace and a column that lists the tools the run should use. The test compares the tools the trace called with that list and reports tool precision, recall, and F1. You can also list distractor tools, which look relevant but are the wrong choice, and track how often your agent picks them.

Taxonomy

  • Task types: LLM.
  • Availability: and .

Why it matters

  • An agent that skips a tool answers without the data it needs, and an agent that calls extra tools adds latency, cost, and risk. Recall catches the first problem and precision the second.
  • Distractor tools surface a specific routing mistake: reaching for a tool that looks right for the request but is not.
  • The test checks the agent’s decisions directly, so you catch a tool-routing regression even when the final answer still looks plausible.

Required columns

To compute this test, each row needs:
  • Trace: The agent run’s trace. The test reads the steps of type tool and names each tool by its function name, or by the step name when there is no function name. In monitoring, instrument your code to send traces. In development, the Openlayer SDK records each row’s trace when the run method of your run script calls traced code.
  • Expected tools: A column that lists the tools the row should use, as a JSON list (["search", "lookup"]) or comma-separated names (search, lookup). Use [] when the row should use no tool.
In monitoring, send the expected tools column as a string, such as '["search", "lookup"]', not as a list. A list-valued column does not reach the test, so the test reports the column as not found and is skipped. In Python, json.dumps(["search", "lookup"]) produces the string. You can attach the column with trace-level metadata or add it to existing traces later.
A tool that is not traced as a tool step looks like the agent used nothing, so every row that expects it fails recall. Framework integrations such as LangChain and the OpenAI Agents SDK record tool calls as tool steps. When you trace your own tool functions, set the step type: @trace(step_type="tool") in Python, or trace(fn, StepType.TOOL) in TypeScript, with StepType imported from openlayer/lib/tracing/steps.

How it works

For every row, the test compares the set of tools the trace called with the set the row expects. Tool names match regardless of case and surrounding spaces, and each tool counts once per row, however many times the agent called it.
  • Precision is the share of the tools used that were expected.
  • Recall is the share of the expected tools that were used.
  • F1 is the harmonic mean of precision and recall.
  • Distractor tool rate is the share of the tools used that are distractor tools.
A row that uses no tool and expects none scores 1 on precision, recall, and F1. Precision is undefined for any other row that used no tool, and recall is undefined for any other row that expects none. With macro averaging, those rows are left out of that measurement’s average. Their F1 is 0. The Averaging setting decides how rows combine:
  • macro (default) scores each row and then averages the rows, so every row counts equally.
  • micro pools the tool matches from all rows before computing the scores, so rows that use or expect more tools count more.
Rows without trace data, and rows whose expected tools value cannot be read, such as an empty cell, are left out of the score.

Example

Take five rows, with weather as a distractor tool: With macro averaging, tool precision is (1 + 1/3 + 1 + 0) / 4 ≈ 0.583, tool recall is (1 + 1/2 + 1 + 0) / 4 = 0.625, and tool F1 is 2.4 / 5 = 0.48. The distractor tool rate averages rows 1, 2, and 4, the rows that used a tool: (0 + 1/3 + 0) / 3 ≈ 0.111. With micro averaging, the rows pool 3 correct tools, 3 unexpected tools, and 2 missed tools. Tool precision is 3 / 6 = 0.5, tool recall is 3 / 5 = 0.6, tool F1 is 6 / 11 ≈ 0.545, and the distractor tool rate is 1 / 6 ≈ 0.167.
The test is skipped, not failed, when the expected tools column is missing, when no row has trace data, when no row has a valid expected tools value, or when no row called a tool and no row expects one. It is also skipped when the selected measurement is undefined for the rows evaluated, such as the distractor tool rate when no row used a tool. The skip message says which.

Parameters

Measurements

Thresholds are set on one of the measurements below. Every value is between 0 and 1.

Test configuration examples

If you are writing a tests.json, here are a few valid configurations for the tool selection accuracy test: