Skip to main content
Evaluation and delay windows are part of every monitoring test. In this guide, we explain their use and highlight their differences. You configure both under Evaluation windows, in the Data section of the test configuration. See Tests configuration for the other sections of a test.

​
Evaluation windows

To evaluate Openlayer tests, a dataset is always required. While in development, the training and validation sets are the natural choices, for monitoring, the dataset used by tests is defined by an evaluation window.
What is the evaluation window?The evaluation window defines the period used to accumulate data, and form the dataset used to evaluate a test.
The Cadence determines how often the window closes and the test runs:
  • Scheduled runs the test on a fixed schedule, such as Daily. Each run uses the data published since the previous run.
  • Manual runs the test only when you trigger it, with Run test from the test’s overflow menu.
Let’s look at a concrete example to clarify the process:
  1. Imagine we want to monitor Nulls on the feature Age for our production data. We would navigate to the test creation page and click the Missing values test.
  2. In the Data section, we set the cadence to Daily.
  3. After the test is successfully created, the platform starts accumulating production data being published.
  4. Then, once the first day passes, it uses the data accumulated in that day, evaluates the number of missing values, and updates the test status.
Each test can have its own cadence. This is important because each value monitored has its peculiarities. For instance, longer evaluation windows can smooth out seasonal data. On the other hand, shorter windows can be appropriate if quick reactions to sudden changes are needed. Two toggles change what each run looks at:
  • Include historical data in each run widens every run to also cover data published before the current window, instead of only the data accumulated since the previous run.
  • Anchor evaluation windows pins the windows to a fixed point in time. Turn it on and you set a Start date and a Timezone — for example, Sep 1, 2026, 12:00 PM in America/Los_Angeles. Every window boundary is then measured from that instant, so a daily test always closes at the same hour of the day you chose.

​
Delay windows

We have seen that evaluation windows are defined to accumulate the data used by tests. Now, we will explore delay windows.
What is the delay window?The delay window defines the gap between the test evaluation time and the end of the evaluation window.
The Evaluation delay toggle is off by default, because most tests can be evaluated at a regular cadence defined solely by the evaluation window. However, in some cases, a delay is also needed. For example, for performance tests, which usually require ground truths to be computed, a delay window is needed because the labels are not available at the same time as the data is published to the platform.