Column statistics

Definition

The column statistics test allows you to set thresholds on statistical measures of individual columns in your dataset. You can select any column and specify a statistic (such as mean, median, variance, etc.), then define acceptable ranges or values for that statistic. This test computes the specified statistical measure for the chosen column and compares it against your defined threshold.

Taxonomy

Task types: LLM, tabular classification, tabular regression.
Availability: and .

Why it matters

Column statistics tests help ensure that your data maintains expected statistical properties over time.
They can detect data quality issues, distribution shifts, or unusual patterns in individual features.
These tests are essential for monitoring data consistency and ensuring that model inputs remain within expected ranges.
Statistical validation helps identify potential data pipeline issues or changes in data collection processes.

Available statistics

The following statistical measures are supported:

Statistic	Description	Typical Use Cases
`mean`	Average value of the column	Monitor if average values stay within expected ranges
`median`	Middle value when data is sorted	Detect shifts in central tendency, robust to outliers
`min`	Minimum value in the column	Ensure no values fall below acceptable minimums
`max`	Maximum value in the column	Detect outliers or values exceeding acceptable maximums
`std`	Standard deviation of the column	Monitor data variability and spread
`sum`	Sum of all values in the column	Useful for totals, counts, or aggregate validations
`count`	Number of non-null values	Monitor data completeness
`variance`	Variance of the column values	Alternative measure of data spread

Test configuration examples

If you are writing a tests.json, here are a few valid configurations for the column statistics test:

[
  {
    "name": "Average age within expected range",
    "description": "Ensures the average age in the dataset is greater than 25",
    "type": "integrity",
    "subtype": "columnStatistic",
    "thresholds": [
      {
        "insightName": "columnStatistic",
        "insightParameters": [
          { "name": "column_name", "value": "age" },
          { "name": "statistic", "value": "mean" }
        ],
        "measurement": "columnStatistic",
        "operator": ">",
        "value": 25
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "b4dee7dc-4f15-48ca-a282-63e2c04e0689"
  },
  {
    "name": "Income variance stability check",
    "description": "Ensures income variance doesn't exceed threshold, indicating stable distribution",
    "type": "integrity",
    "subtype": "columnStatistic",
    "thresholds": [
      {
        "insightName": "columnStatistic",
        "insightParameters": [
          { "name": "column_name", "value": "income" },
          { "name": "statistic", "value": "variance" }
        ],
        "measurement": "columnStatistic",
        "operator": "<=",
        "value": 1000000
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "96622fba-ea00-4e42-8f42-5e8f5f60805f"
  }
]

[
  {
    "name": "Transaction amount median monitoring",
    "description": "Monitors median transaction amount to detect unusual patterns",
    "type": "integrity",
    "subtype": "columnStatistic",
    "thresholds": [
      {
        "insightName": "columnStatistic",
        "insightParameters": [
          { "name": "column_name", "value": "transaction_amount" },
          { "name": "statistic", "value": "median" }
        ],
        "measurement": "columnStatistic",
        "operator": ">=",
        "value": 10.0
      },
      {
        "insightName": "columnStatistic",
        "insightParameters": [
          { "name": "column_name", "value": "transaction_amount" },
          { "name": "statistic", "value": "median" }
        ],
        "measurement": "columnStatistic",
        "operator": "<=",
        "value": 500.0
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 3600,
    "delayWindow": 0,
    "syncId": "b4dee7dc-4f15-48ca-a282-63e2c04e0689"
  },
  {
    "name": "Data completeness check",
    "description": "Ensures sufficient non-null values in critical columns",
    "type": "integrity",
    "subtype": "columnStatistic",
    "thresholds": [
      {
        "insightName": "columnStatistic",
        "insightParameters": [
          { "name": "column_name", "value": "customer_id" },
          { "name": "statistic", "value": "count" }
        ],
        "measurement": "columnStatistic",
        "operator": ">=",
        "value": 1000
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 3600,
    "delayWindow": 0,
    "syncId": "96622fba-ea00-4e42-8f42-5e8f5f60805f"
  }
]

Column drift

Column values match

⌘I

Get started

Workspace setup

Governance

Observability

Offline testing

Tests

Gateway

Data quality monitoring

Administration

Notifications

Other resources

Definition

Taxonomy

Why it matters

Available statistics

Test configuration examples

​Definition

​Taxonomy

​Why it matters

​Available statistics

​Test configuration examples

Definition

Taxonomy

Why it matters

Available statistics

Test configuration examples