Skip to main content
Databricks hero Openlayer integrates with Databricks so you can run data quality tests directly on your Databricks tables. The integration uses a personal access token (PAT) tied to a secure Databricks connection. This ensures auditable, key-based access without requiring usernames or passwords.

​
Prerequisites

To follow this guide, you need:
  • A Databricks account and workspace with SQL warehouses enabled
  • Permissions to create and use a personal access token (PAT)
  • A table in Databricks you want to monitor (with timestamp and unique ID columns recommended)
  • An Openlayer project with monitoring mode enabled

​
Setup Guide

​
Step 1: Generate a personal access token

In your Databricks workspace:
  1. Go to User Settings → Developer → Access Tokens.
  2. Click Generate new token.
  3. Copy and store the PAT securely — you will provide it when connecting Openlayer.
See Databricks documentation for details.

​
Step 2: Collect connection details

You will need:
  • Hostname: your workspace URL (e.g. https://dbc-247310bd-93fc.cloud.databricks.com)
  • Port: typically 443
  • SQL Warehouse endpoint: path to the warehouse, e.g. /sql/1.0/warehouses/<warehouse-id>
  • Personal access token (PAT): generated in step 1

​
Step 3: Connect inside Openlayer

In your Openlayer workspace:
  1. Go to Data sources and select Databricks.
  2. Click Connect.
  3. Fill in the fields:
  • Hostname: your workspace hostname (e.g. https://dbc-247310bd-93fc.cloud.databricks.com)
  • Port: usually 443
  • SQL Warehouse endpoint: path to your warehouse
  • Personal access token: PAT you generated
  • Name: a descriptive label for this connection

​
Step 4: Configure your table

After the connection is created, configure the table you want to monitor:
  1. In Databricks catalog, select or enter the catalog that contains your table.
  2. In Databricks schema, select or enter a schema. The available schemas depend on the catalog you selected.
  3. In Databricks tables, select a table. Each option shows the table name and type. Openlayer stores your selection as a fully qualified name, such as samples.nyctaxi.trips or workspace.openlayer_demo.landing_inferences.
  4. Select the timestamp, unique ID, and any optional ML columns from the table’s columns. Each option shows the column name and type, and timestamp-compatible columns appear first in the timestamp field. If the list is empty or a column is missing, you can type its name.
  5. Enter a descriptive Data source name for the table in Openlayer.

​
Optional: ML-specific settings

If the table contains ML outputs, you can provide additional context:
  • Class names
  • Feature names
  • Categorical feature names
  • Predictions column (optional)
This enables Openlayer to run ML-aware tests such as drift detection and performance monitoring.

​
Troubleshooting

  • Authentication errors → verify that your PAT is valid and not expired.
  • Connection errors → confirm the hostname, port, and SQL warehouse endpoint are correct.
  • Catalogs, schemas, tables, or columns do not appear → confirm the connection works and your PAT can access the SQL warehouse and target tables. You can type names manually.
  • Empty results → check that the timestamp column is populated and you’ve selected the correct table.
  • Permission errors → ensure your PAT user has access to the warehouse and the target tables.