> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.
# IBM Watson Orchestrate
> Integrate IBM watsonx Orchestrate with Openlayer for agent tracing, evaluation, and monitoring
Openlayer integrates with [IBM watsonx Orchestrate](https://www.ibm.com/products/watsonx-orchestrate) to help you trace, evaluate, and monitor your deployed agents.
The integration works by pulling agent execution traces directly from watsonx Orchestrate's Observability (AgentOps) API and automatically mapping them into structured Openlayer traces — LLM calls, tool executions, latency, token usage, and cost — grouped by conversation session.
## How it works
Openlayer connects to your watsonx Orchestrate instance and automatically:
* Discovers the agents deployed in your instance
* Pulls each enabled agent's run traces on a schedule
* Maps every trace into structured steps (LLM calls, tool calls, and spans)
* Captures session and user identifiers, token usage, cost, and latency
* Writes each trace into a linked Openlayer project and inference pipeline
The integration is **pull-based** — Openlayer reaches out to watsonx Orchestrate, so it never requires inbound access to your Openlayer instance. This makes it a good fit for on-premise and airgapped deployments.
***
## Prerequisites
To connect watsonx Orchestrate, you need:
* An **API key** for your watsonx Orchestrate instance
* The instance **base URL** (for example, `https://api..watson-orchestrate.ibm.com/instances/`)
Authentication uses **MCSP** (the Multi-Cloud SaaS Platform token exchange),
not IBM Cloud IAM. Openlayer exchanges your API key for a short-lived bearer
token automatically — you only need to provide the API key.
On-premise **Cloud Pak for Data (CP4D)** deployments are supported but the
auth flow is currently **experimental** and not yet verified against a live
Cloud Pak instance.
***
## Set up the integration
The native integration connects Openlayer directly to your watsonx Orchestrate instance. Openlayer discovers your agents, pulls their traces on a schedule, and links each to a project — all managed from the Openlayer UI.
### Step 1: Connect your instance
1. In Openlayer, navigate to **Settings** → **Integrations**.
2. Click **IBM Watson Orchestrate**.
3. Choose your **deployment type** (SaaS or on-premise / CP4D) and fill in:
* **Base URL** — your instance API base URL
* **API key**
* For on-premise: **Instance ID** and, optionally, a custom **token endpoint (IAM URL)**
4. Click **Connect**.
Openlayer verifies your credentials by requesting a token from the MCSP token endpoint. If successful, the connection status changes to **Connected**.
### Step 2: Discover your agents
Once connected, click **Discover agents** to scan your instance. Openlayer lists every agent deployed in watsonx Orchestrate.
Each discovered agent shows its:
* **Name** and backing **LLM**
* **Status** in watsonx Orchestrate (active or inactive)
* **Monitoring status** in Openlayer (enabled, disabled, or error)
### Step 3: Enable monitoring
For each agent you want to monitor, click **Enable**. You have two options:
* **Auto-create a new project** — Openlayer creates a project named
`Watson Orchestrate - ` with a default inference pipeline. This is
the fastest way to get started.
* **Map to an existing project** — Choose an existing Openlayer project and
inference pipeline. Use this when you already have a project set up for the
agent.
Turn on **Auto-enable new agents** in the connection settings to automatically
enable monitoring for every agent Openlayer discovers. This is useful when you
want all agents in an instance monitored by default.
### Step 4: Configure sync settings
After enabling at least one agent, configure how often Openlayer pulls new traces:
* **Sync enabled** — Toggle periodic syncing on or off.
* **Sync frequency** — How often Openlayer checks for new traces
(default: every 60 minutes).
* **Sync range** — Choose between:
* **All available data** — Sync all historical traces.
* **Last 7 days** — Only sync recent runs.
* **Custom date** — Specify a start date.
You can also trigger a **manual sync** at any time by clicking **Sync now**.
### Backfilling historical data
To re-sync historical data for a specific agent (for example, after adjusting
evaluation tests), open the three-dot menu on the agent row and select
**Backfill**. You can backfill:
* **All available data** — Re-process every run for this agent.
* **Since a specific date** — Only re-process runs from the given date onward.
Backfill is idempotent — re-syncing a run that Openlayer has already ingested
will not create duplicate traces.
***
## Monitoring in Openlayer
Once integrated, watsonx Orchestrate traces will automatically appear in the linked Openlayer project.
### View agent traces
Navigate to your project's **Records** tab to see detailed traces.
Each trace captures:
* The user query and the agent's response
* Nested execution steps (LLM calls, tool executions, and spans)
* Latency, token usage, and cost per step
* The conversation **session** the run belongs to
### Monitor session outcomes
Because traces are grouped by session, you can track conversation-level metrics
across multi-turn interactions — turn counts, latency, cost, and any
session-level scores you configure.
### Run evaluations
Create evaluation pipelines to:
* Score response quality
* Detect hallucinations
* Track safety and compliance
* Measure tool-use correctness