Freshness

​

Definition

​

Taxonomy

• Task types: LLM, tabular classification, tabular regression, text classification.
• Availability: monitoring only.

​

Why it matters

• A data source that stops receiving data fails silently: most tests skip a window with no data, so nothing alerts you that traffic or a pipeline stopped.
• The freshness test turns “no data for too long” into a failing test, so you hear about an outage, a broken integration, or a stalled job as soon as it exceeds your tolerance.
• It needs no knowledge of your schema, which makes it a good default test on every monitored data source.

​

Parameters

• Unit: the time unit the age and the threshold are expressed in — minutes, hours, days, or weeks. The age is fractional, so with hours, 90 minutes is 1.5.

​

How windows are evaluated

• Empty windows still evaluate. The age comes from the data source’s ingestion history, not from the rows in the window, so a window with no data reports how long it has been since data last arrived.
• Windows before any data arrived are skipped with “No data received yet.” When the test has filters, the message is “No data matching the filters received yet.”
• The age reflects when Openlayer received the data, not timestamps inside it. To measure how recent the values in a datetime column are, use the column freshness test.

​

Automatic thresholds

• The upper bound is never lower than one evaluation window. A data source that streams constantly sits near an age of zero, and a bound that tight would fail on every short gap. For example, with a daily evaluation window and hours as the unit, the upper bound is at least 24.
• The test fails once a whole window passes without data, even before there is enough history to forecast. Until then, the upper bound is one evaluation window, instead of the test passing by default.
• An ongoing outage doesn’t teach the forecast to expect it. The windows since the data source last received data are left out of the history, so the bound doesn’t grow along with the outage. Earlier gaps stay in, so a data source that syncs on a schedule still learns its rhythm.

​

Unsupported data sources

​

Test configuration examples

[
  {
    "name": "Data source keeps receiving data",
    "description": "Fails when the age of the latest data exceeds the learned upper bound",
    "type": "integrity",
    "subtype": "freshness",
    "thresholds": [
      {
        "insightName": "freshness",
        "insightParameters": [{ "name": "unit", "value": "hours" }],
        "measurement": "freshness",
        "operator": "<=",
        "thresholdMode": "automatic"
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 3600,
    "delayWindow": 0,
    "syncId": "b4dee7dc-4f15-48ca-a282-63e2c04e0689"
  }
]

[
  {
    "name": "Data received within the last 6 hours",
    "description": "Fails when the data source hasn't received data for more than 6 hours",
    "type": "integrity",
    "subtype": "freshness",
    "thresholds": [
      {
        "insightName": "freshness",
        "insightParameters": [{ "name": "unit", "value": "hours" }],
        "measurement": "freshness",
        "operator": "<=",
        "thresholdMode": "manual",
        "value": 6
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 3600,
    "delayWindow": 0,
    "syncId": "96622fba-ea00-4e42-8f42-5e8f5f60805f"
  }
]

​

Related

• Column freshness — how recent the values in a datetime column are, available in development and monitoring.
• Number of rows — how much data arrived in a window, rather than when it last arrived.
• Evaluation and delay windows — how windows are defined.