Column freshness

Definition

The column freshness test measures how old the most recent or oldest value of a datetime column is. The result is shown as Age, in the unit you choose, and you set a threshold on it — for example, that the most recent event_time is at most 6 hours old. In monitoring, Openlayer measures the age relative to the end of each evaluation window, not when the test runs, so re-running a past window gives the same result.

Taxonomy

Task types: LLM, tabular classification, tabular regression, text classification.
Availability: and .

Why it matters

Data can keep arriving while its contents go stale — for example, a job that re-sends yesterday’s snapshot, or a table whose upstream source stopped updating.
Measuring the timestamps inside the data catches staleness that row counts and arrival times miss.
It works on tables connected from a data warehouse, where the freshness test isn’t supported.

Parameters

Parameter	Values	Description
`column_name`	Any datetime column	The column to measure.
`aggregation`	`max`, `min`	Shown as Measure in the UI. `max` (Most recent value) catches data that stopped flowing. `min` (Oldest value) catches stale rows that should have been replaced.
`unit`	`minutes`, `hours`, `days`, `weeks`	The time unit the age and the threshold are expressed in. The age is fractional, so with `hours`, 90 minutes is `1.5`.

The column must contain dates, timestamps, or Unix epochs in seconds or milliseconds. Timestamps without a timezone are read as UTC. If the column holds other values, or has no values at all, the test is skipped. In monitoring, the test only looks at the rows in the evaluation window, after applying any filters.

How windows are evaluated

Empty windows fail. A window with no data usually means data stopped arriving, so the test fails with “No data received during the evaluation window.” When the test has filters and no rows match them, the message mentions the filters.
Windows before the data source existed are skipped with “No data received yet.”

Reserved column names

In monitoring, created_at and updated_at are reserved: Openlayer sets them to the time it ingests each row. If your data has a column with either name, it is overwritten, so the column freshness test can’t measure your own values in it. Publish those timestamps under a different name, such as event_updated_at, and measure that column instead. To alert on when data last arrived, use the freshness test.

Test configuration examples

If you are writing a tests.json, here are a few valid configurations for the column freshness test:

[
  {
    "name": "Dataset refreshed within the last 7 days",
    "description": "Asserts that the most recent snapshot_date is at most 7 days old",
    "type": "integrity",
    "subtype": "columnFreshness",
    "thresholds": [
      {
        "insightName": "columnFreshness",
        "insightParameters": [
          { "name": "column_name", "value": "snapshot_date" },
          { "name": "aggregation", "value": "max" },
          { "name": "unit", "value": "days" }
        ],
        "measurement": "freshness",
        "operator": "<=",
        "thresholdMode": "manual",
        "value": 7
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "b4dee7dc-4f15-48ca-a282-63e2c04e0689"
  }
]

[
  {
    "name": "Latest event is recent",
    "description": "Fails when the most recent event_time is older than the learned upper bound",
    "type": "integrity",
    "subtype": "columnFreshness",
    "thresholds": [
      {
        "insightName": "columnFreshness",
        "insightParameters": [
          { "name": "column_name", "value": "event_time" },
          { "name": "aggregation", "value": "max" },
          { "name": "unit", "value": "hours" }
        ],
        "measurement": "freshness",
        "operator": "<=",
        "thresholdMode": "automatic"
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 3600,
    "delayWindow": 0,
    "syncId": "b4dee7dc-4f15-48ca-a282-63e2c04e0689"
  },
  {
    "name": "No rows older than 30 days",
    "description": "Fails when the oldest event_time in the window is more than 30 days old",
    "type": "integrity",
    "subtype": "columnFreshness",
    "thresholds": [
      {
        "insightName": "columnFreshness",
        "insightParameters": [
          { "name": "column_name", "value": "event_time" },
          { "name": "aggregation", "value": "min" },
          { "name": "unit", "value": "days" }
        ],
        "measurement": "freshness",
        "operator": "<=",
        "thresholdMode": "manual",
        "value": 30
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 86400,
    "delayWindow": 0,
    "syncId": "96622fba-ea00-4e42-8f42-5e8f5f60805f"
  }
]

Freshness — how long ago the data source last received data, regardless of the timestamps inside it.
Column statistics — other statistics on a single column.
Evaluation and delay windows — how windows are defined.

Get started

Workspace setup

Governance

Observability

Offline testing

Tests

Gateway

Data quality monitoring

Administration

Notifications

Other resources

Definition

Taxonomy

Why it matters

Parameters

How windows are evaluated

Reserved column names

Test configuration examples

​Definition

​Taxonomy

​Why it matters

​Parameters

​How windows are evaluated

​Reserved column names

​Test configuration examples

​Related

Definition

Taxonomy

Why it matters

Parameters

How windows are evaluated

Reserved column names

Test configuration examples

Related