The freshness test measures how long before the end of the evaluation window the data
source last received data. If you add filters to the test, only data matching them counts.
The result is shown as Age, in the unit you choose, and you set a threshold on it — for
example, that the data source received data within the last 6 hours.Openlayer measures the age at the end of each
evaluation window, not when the test runs, so
re-running a past window gives the same result.
A data source that stops receiving data fails silently: most tests skip a window with no
data, so nothing alerts you that traffic or a pipeline stopped.
The freshness test turns “no data for too long” into a failing test, so you hear about an
outage, a broken integration, or a stalled job as soon as it exceeds your tolerance.
It needs no knowledge of your schema, which makes it a good default test on every
monitored data source.
Unit: the time unit the age and the threshold are expressed in — minutes, hours,
days, or weeks. The age is fractional, so with hours, 90 minutes is 1.5.
Empty windows still evaluate. The age comes from the data source’s ingestion history,
not from the rows in the window, so a window with no data reports how long it has been
since data last arrived.
Windows before any data arrived are skipped with “No data received yet.” When the test
has filters, the message is “No data matching the filters received yet.”
The age reflects when Openlayer received the data, not timestamps inside it. To measure
how recent the values in a datetime column are, use the
column freshness test.
The freshness test supports automatic thresholds. Use an upper
bound — Upper only in the UI, or operator <= in tests.json — since a growing age is
what signals a problem. Freshness adjusts automatic thresholds in three ways:
The upper bound is never lower than one evaluation window. A data source that streams
constantly sits near an age of zero, and a bound that tight would fail on every short gap.
For example, with a daily evaluation window and hours as the unit, the upper bound is at
least 24.
The test fails once a whole window passes without data, even before there is enough
history to forecast. Until then, the upper bound is one evaluation window, instead of the
test passing by default.
An ongoing outage doesn’t teach the forecast to expect it. The windows since the data
source last received data are left out of the history, so the bound doesn’t grow along
with the outage. Earlier gaps stay in, so a data source that syncs on a schedule still
learns its rhythm.
The freshness test relies on Openlayer recording when each row arrives. On data sources where
that isn’t recorded, such as tables connected from a data warehouse, the test is skipped with
“Freshness isn’t supported for this data backend. Use column freshness to measure a datetime
column instead.” Use the column freshness test on a column
that records when each row was written.
