Skip to main content
Each Openlayer test can be tailored to your needs. You can choose the data to run on, adjust thresholds, tweak parameters, and more. Most options have sensible defaults, so setup is usually minimal. This page walks you through all available configurations. Test configuration

​
Configurations

The configurations available are:
The “Data” section allows you to select the data on top of which this test will run.In monitoring mode, it means choosing among your data sources (e.g., production data, staging data, etc.) In development mode, it means selecting among your datasets (e.g., training, validation, etc.)You can also apply filters to narrow the test to specific slices of the data, and turn on Apply to new data sources so the test covers data sources you connect later. When you fill in a filter, the value picker suggests values from your data — see Filter value suggestions.In monitoring mode, the section also holds the Evaluation windows settings: the Cadence the test runs on (Scheduled or Manual), plus the Evaluation delay, Include historical data in each run, and Anchor evaluation windows toggles. See Evaluation and delay windows for details.
Some tests include configurable parameters that control their behavior. For example, a test such as the PII detection test lets you specify which type of PII to detect (e.g., credit card numbers, social security numbers, etc.)Custom metrics can also declare their own configurable parameters. See Configurable parameters for details on how to define them.Refer to the specific test page in the Catalog for details on its parameters.For LLM-as-a-judge tests, the Judge settings hold the judge model and the sampling controls that bound how much data the judge scores: Sample size (%) (100 by default), Data limit (rows) (50 by default), and the Sampling method (for example, Random). You can set the same controls for every test in the project under Settings → LLM-as-a-judge.
The “Threshold” section allows you to define the condition for the test to pass.On the Manual tab, you pick a Measurement, an Operator, and a Threshold value — saying, for example, that the test passes if the metric is greater than 0.5.In monitoring mode, most tests also offer an Auto tab. With automatic thresholds, Openlayer automatically learns the distribution of historical data and predicts upper and lower bounds for the metric. The test fails if the observed value falls outside the predicted range.Automatic thresholds are directional: choose Both bounds to fail on either side of the predicted range, or Upper only or Lower only to fail on one side only — useful when a move in a single direction is what concerns you, such as a rising cost.
The “Advanced settings” section holds the test metadata. Here’s where you set the test’s Priority.Cadence, delays, and anchoring live in the “Data” section — see Evaluation and delay windows.

​
Filter value suggestions

When you filter a test on a column, the value picker suggests values Openlayer has already seen in that column, so you can select a value instead of typing it from memory. Every filter value picker can show these suggestions, not just semantic search filters, and the kind of comparison you are making determines which values it suggests — whether you are matching on equality, membership, contains, or an ordering comparison. The same suggestions appear when you filter monitoring records: the Records table’s filter prompts you to “Enter value…”, then lists matching values under “In the last 30 days” with each value’s count beside it. Each suggested value carries a count tag. Hover it to see how many rows matched, as in “12 matches found in recent data”, together with the note “Suggestions are drawn from a recent sample of rows, so the full dataset may contain more matches.” Read the count as a signal of whether a value is worth filtering on rather than as an exact total. Columns with many distinct values do not list suggestions up front. The picker asks you to “Type to search values” instead, and looks up matching values as you type.

​
Pausing tests

In monitoring mode, you can pause tests when you don’t want them to compute results — for example, while you are replaying data or waiting on an upstream fix. Open the overflow menu on a test row and select Pause. While tests are paused, the tests list shows the banner “Tests are paused. You can still create tests, but they will not compute results.” Select Unpause in the banner to start receiving test results again.