> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Tool selection accuracy

> Learn how to use the tool selection accuracy test to check whether your agent calls the tools each request needs, no more and no fewer

## Definition

The tool selection accuracy test checks whether your agent calls the right
tools: every tool a request needs, and nothing else. Each row is one agent run,
with its trace and a column that lists the tools the run should use. The test
compares the tools the trace called with that list and reports tool precision,
recall, and F1.

You can also list **distractor tools**, which look relevant but are the wrong
choice, and track how often your agent picks them.

## Taxonomy

* **Task types**: LLM.
* **Availability**: development (Continuously evaluate your models and datasets as you iterate on their versions.)
  and monitoring (Monitor a model in production, measure its health, check for drifts and set up alerts.).

## Why it matters

* An agent that skips a tool answers without the data it needs, and an agent
  that calls extra tools adds latency, cost, and risk. Recall catches the first
  problem and precision the second.
* Distractor tools surface a specific routing mistake: reaching for a tool that
  looks right for the request but is not.
* The test checks the agent's decisions directly, so you catch a tool-routing
  regression even when the final answer still looks plausible.

## Required columns

To compute this test, each row needs:

* **Trace**: The agent run's trace. The test reads the steps of type `tool` and
  names each tool by its function name, or by the step name when there is no
  function name. In monitoring, [instrument your code](https://docs.openlayer.com/monitoring/instrument.md)
  to send traces. In development, the Openlayer SDK records each row's trace
  when the `run` method of your
  [run script](https://docs.openlayer.com/development/configuring-output-generation.md#the-run-script)
  calls traced code.
* **Expected tools**: A column that lists the tools the row should use, as a
  JSON list (`["search", "lookup"]`) or comma-separated names
  (`search, lookup`). Use `[]` when the row should use no tool.

> [!WARNING]
>
> In monitoring, send the expected tools column as a **string**, such as
> `'["search", "lookup"]'`, not as a list. A list-valued column does not reach
> the test, so the test reports the column as not found and is skipped. In
> Python, `json.dumps(["search", "lookup"])` produces the string. You can attach
> the column with [trace-level
> metadata](https://docs.openlayer.com/monitoring/metadata.md#trace-level-metadata) or [add it to existing
> traces](https://docs.openlayer.com/monitoring/updating-data.md) later.

> [!TIP]
>
> A tool that is not traced as a `tool` step looks like the agent used nothing,
> so every row that expects it fails recall. Framework integrations such as
> [LangChain](https://docs.openlayer.com/integrations/langchain.md) and the [OpenAI Agents
> SDK](https://docs.openlayer.com/integrations/openai-agents-sdk.md) record tool calls as tool steps. When
> you trace your own tool functions, set the step type:
> `@trace(step_type="tool")` in Python, or `trace(fn, StepType.TOOL)` in
> TypeScript, with `StepType` imported from `openlayer/lib/tracing/steps`.

## How it works

For every row, the test compares the set of tools the trace called with the set
the row expects. Tool names match regardless of case and surrounding spaces, and
each tool counts once per row, however many times the agent called it.

* **Precision** is the share of the tools used that were expected.
* **Recall** is the share of the expected tools that were used.
* **F1** is the harmonic mean of precision and recall.
* **Distractor tool rate** is the share of the tools used that are distractor
  tools.

A row that uses no tool and expects none scores 1 on precision, recall, and F1.
Precision is undefined for any other row that used no tool, and recall is
undefined for any other row that expects none. With `macro` averaging, those
rows are left out of that measurement's average. Their F1 is 0.

The **Averaging** setting decides how rows combine:

* `macro` (default) scores each row and then averages the rows, so every row
  counts equally.
* `micro` pools the tool matches from all rows before computing the scores, so
  rows that use or expect more tools count more.

Rows without trace data, and rows whose expected tools value cannot be read,
such as an empty cell, are left out of the score.

### Example

Take five rows, with `weather` as a distractor tool:

| Row | Tools used | Expected tools | Precision | Recall | F1 |
| - | - | - | - | - | - |
| 1 | `search` (twice), `lookup` | `["Search", "lookup"]` | 1 | 1 | 1 |
| 2 | `search`, `weather`, `calc` | `search, lookup` | 1/3 | 1/2 | 0.4 |
| 3 | None | `[]` | 1 | 1 | 1 |
| 4 | `search` | `[]` | 0 | — | 0 |
| 5 | None | `lookup` | — | 0 | 0 |

With `macro` averaging, tool precision is (1 + 1/3 + 1 + 0) / 4 ≈ 0.583, tool
recall is (1 + 1/2 + 1 + 0) / 4 = 0.625, and tool F1 is 2.4 / 5 = 0.48. The
distractor tool rate averages rows 1, 2, and 4, the rows that used a tool:
(0 + 1/3 + 0) / 3 ≈ 0.111.

With `micro` averaging, the rows pool 3 correct tools, 3 unexpected tools, and
2 missed tools. Tool precision is 3 / 6 = 0.5, tool recall is 3 / 5 = 0.6, tool
F1 is 6 / 11 ≈ 0.545, and the distractor tool rate is 1 / 6 ≈ 0.167.

> [!NOTE]
>
> The test is skipped, not failed, when the expected tools column is missing,
> when no row has trace data, when no row has a valid expected tools value, or
> when no row called a tool and no row expects one. It is also skipped when the
> selected measurement is undefined for the rows evaluated, such as the
> distractor tool rate when no row used a tool. The skip message says which.

## Parameters

| Parameter | Required | Default | Description |
| - | - | - | - |
| `expected_tools_column` | Yes | | **Expected tools column**. The column that lists the tools each row should use. |
| `distractor_tools` | No | `[]` | **Distractor tools**. Up to 100 tool names that look relevant but are the wrong choice, such as `["weather"]`. Needed for the distractor tool rate. |
| `averaging` | No | `macro` | **Averaging**. `macro` averages the row scores; `micro` pools the tool matches across rows. |

## Measurements

Thresholds are set on one of the measurements below. Every value is between 0
and 1.

| Measurement | Description |
| - | - |
| `toolPrecision` | **Tool precision**. Share of the tools used that were expected. Higher is better, so use `>=` or `>`. |
| `toolRecall` | **Tool recall**. Share of the expected tools that were used. Higher is better, so use `>=` or `>`. |
| `toolF1` | **Tool F1**. Harmonic mean of tool precision and recall. Higher is better, so use `>=` or `>`. |
| `distractorToolRate` | **Distractor tool rate**. Share of the tools used that are distractor tools. Lower is better, so use `<=` or `<`. Requires `distractor_tools`. |

## Test configuration examples

If you are writing a `tests.json`, here are a few valid configurations for the tool selection accuracy test:

```json Development theme={null}
[
  {
    "name": "Tool F1 at least 0.8",
    "description": "Ensures the agent calls the tools each row expects, with a tool F1 of at least 0.8",
    "type": "integrity",
    "subtype": "toolSelectionAccuracy",
    "thresholds": [
      {
        "insightName": "toolSelectionAccuracy",
        "insightParameters": [
          { "name": "expected_tools_column", "value": "expected_tools" },
          { "name": "averaging", "value": "macro" }
        ],
        "measurement": "toolF1",
        "operator": ">=",
        "value": 0.8
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "593326db-6b41-4ca5-bb4c-83d73905282e"
  }
]
```

```json Monitoring theme={null}
[
  {
    "name": "Distractor tool rate at most 0.05",
    "description": "Ensures that at most 5% of the tools the agent calls are distractor tools",
    "type": "integrity",
    "subtype": "toolSelectionAccuracy",
    "thresholds": [
      {
        "insightName": "toolSelectionAccuracy",
        "insightParameters": [
          { "name": "expected_tools_column", "value": "expected_tools" },
          { "name": "distractor_tools", "value": ["weather"] },
          { "name": "averaging", "value": "micro" }
        ],
        "measurement": "distractorToolRate",
        "operator": "<=",
        "value": 0.05
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 3600,
    "delayWindow": 0,
    "syncId": "9abc7db7-9256-4e92-81e2-752ac575fd76"
  }
]
```

## Related

* [Unauthorized tool calls test](https://docs.openlayer.com/tests/catalog/unauthorized-tool-calls.md) - Check that an agent only calls tools from an allowed list.
* [Instrument your code](https://docs.openlayer.com/monitoring/instrument.md) - Send traces from your agent to Openlayer.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.