Skip to main content
LangChain hero Openlayer integrates with IBM watsonx Orchestrate to help you trace, evaluate, and monitor your deployed agents. The integration works by pulling agent execution traces directly from watsonx Orchestrate’s Observability (AgentOps) API and automatically mapping them into structured Openlayer traces — LLM calls, tool executions, latency, token usage, and cost — grouped by conversation session.

​
How it works

Openlayer connects to your watsonx Orchestrate instance and automatically:
  • Discovers the agents deployed in your instance
  • Pulls each enabled agent’s run traces on a schedule
  • Maps every trace into structured steps (LLM calls, tool calls, and spans)
  • Captures session and user identifiers, token usage, cost, and latency
  • Writes each trace into a linked Openlayer project and inference pipeline
The integration is pull-based — Openlayer reaches out to watsonx Orchestrate, so it never requires inbound access to your Openlayer instance. This makes it a good fit for on-premise and airgapped deployments.

​
Prerequisites

To connect watsonx Orchestrate, you need:
  • An API key for your watsonx Orchestrate instance
  • The instance base URL (for example, https://api.<region>.watson-orchestrate.ibm.com/instances/<instance-id>)
Authentication uses MCSP (the Multi-Cloud SaaS Platform token exchange), not IBM Cloud IAM. Openlayer exchanges your API key for a short-lived bearer token automatically — you only need to provide the API key.
On-premise Cloud Pak for Data (CP4D) deployments are supported but the auth flow is currently experimental and not yet verified against a live Cloud Pak instance.

​
Set up the integration

The native integration connects Openlayer directly to your watsonx Orchestrate instance. Openlayer discovers your agents, pulls their traces on a schedule, and links each to a project — all managed from the Openlayer UI.

​
Step 1: Connect your instance

  1. In Openlayer, navigate to Settings → Integrations.
  2. Click IBM Watson Orchestrate.
  3. Choose your deployment type (SaaS or on-premise / CP4D) and fill in:
    • Base URL — your instance API base URL
    • API key
    • For on-premise: Instance ID and, optionally, a custom token endpoint (IAM URL)
  4. Click Connect.
Openlayer verifies your credentials by requesting a token from the MCSP token endpoint. If successful, the connection status changes to Connected.

​
Step 2: Discover your agents

Once connected, click Discover agents to scan your instance. Openlayer lists every agent deployed in watsonx Orchestrate. Each discovered agent shows its:
  • Name and backing LLM
  • Status in watsonx Orchestrate (active or inactive)
  • Monitoring status in Openlayer (enabled, disabled, or error)

​
Step 3: Enable monitoring

For each agent you want to monitor, click Enable. You have two options:
  • Auto-create a new project — Openlayer creates a project named Watson Orchestrate - <agent name> with a default inference pipeline. This is the fastest way to get started.
  • Map to an existing project — Choose an existing Openlayer project and inference pipeline. Use this when you already have a project set up for the agent.
Turn on Auto-enable new agents in the connection settings to automatically enable monitoring for every agent Openlayer discovers. This is useful when you want all agents in an instance monitored by default.

​
Step 4: Configure sync settings

After enabling at least one agent, configure how often Openlayer pulls new traces:
  • Sync enabled — Toggle periodic syncing on or off.
  • Sync frequency — How often Openlayer checks for new traces (default: every 60 minutes).
  • Sync range — Choose between:
    • All available data — Sync all historical traces.
    • Last 7 days — Only sync recent runs.
    • Custom date — Specify a start date.
You can also trigger a manual sync at any time by clicking Sync now.

​
Backfilling historical data

To re-sync historical data for a specific agent (for example, after adjusting evaluation tests), open the three-dot menu on the agent row and select Backfill. You can backfill:
  • All available data — Re-process every run for this agent.
  • Since a specific date — Only re-process runs from the given date onward.
Backfill is idempotent — re-syncing a run that Openlayer has already ingested will not create duplicate traces.

​
Monitoring in Openlayer

Once integrated, watsonx Orchestrate traces will automatically appear in the linked Openlayer project.

​
View agent traces

Navigate to your project’s Records tab to see detailed traces. Each trace captures:
  • The user query and the agent’s response
  • Nested execution steps (LLM calls, tool executions, and spans)
  • Latency, token usage, and cost per step
  • The conversation session the run belongs to

​
Monitor session outcomes

Because traces are grouped by session, you can track conversation-level metrics across multi-turn interactions — turn counts, latency, cost, and any session-level scores you configure.

​
Run evaluations

Create evaluation pipelines to:
  • Score response quality
  • Detect hallucinations
  • Track safety and compliance
  • Measure tool-use correctness