Group metric parity

Definition

The group metric parity test compares one metric across protected groups, such as each gender or age band. You can compare model-quality metrics such as AUC, log loss, MAE, or R², or the mean, median, p95, or rate of a column such as a judge score, latency, cost, refusal flag, or tool-error flag. For every group, the test computes the metric’s value, its direction-aware ratio to a comparison point, and its gap from that point. A ratio below 1 always means that the group is worse, whether higher or lower values are better. You can compare each group with a reference group or with the metric over all rows, and you can combine up to 3 protected attributes to test intersections.

Taxonomy

Task types: LLM, tabular classification, tabular regression, text classification.
Availability: and .

Pipelines backed by BigQuery or Snowflake are not yet supported by the fairness tests. The test is skipped on them.

Why it matters

Aggregate quality can hide a model that performs well overall but has a much higher error or loss for one group.
The column metrics extend parity checks beyond model accuracy. For example, you can compare judge scores, latency, cost, refusals, or tool errors to find a quality-of-service gap that affects only one group.
The ratio gives you a scale-independent screen, while the gap shows the shortfall in the metric’s own units.
The small-group guard and significance options help you distinguish a repeatable disparity from noise in a short evaluation window.
On a monitoring schedule, the test catches group-level changes before they are visible in the overall metric.

Metrics by task type

The available metrics depend on the project’s task type:

Task type	Available metrics
LLM	AUC, log loss, Brier score, NPV, false omission rate, MAE, RMSE, R², mean, median, p95, and rate.
Tabular classification and text classification	AUC, log loss, Brier score, NPV, false omission rate, mean, median, p95, and rate.
Tabular regression	MAE, RMSE, R², mean, median, p95, and rate.

The classification metrics use a ground-truth column and the positive values you select. AUC reads a score column; log loss and Brier score read a predicted probability; NPV and false omission rate read a prediction column. MAE, RMSE, and R² read numeric prediction and ground-truth columns. The mean, median, p95, and rate options can use any suitable top-level column. For rate, you also select the values that count as positive. For all four column metrics, Direction is required and has no default: choose whether a higher or a lower value is better.

How it works

The test follows these steps on the rows of each evaluation window:

Drop incomplete rows. Rows missing an input needed by the selected metric are excluded first. Among rows with the metric inputs, rows with a missing value in any protected attribute column are excluded. A blank or whitespace-only attribute counts as missing. missingAttributeRate is the share of rows with the metric inputs that were excluded for a missing attribute.
Form groups and set small ones aside. Rows are grouped by the distinct values of the protected attribute columns. A group is sufficient when it has at least min_group_size rows and at least min_group_share of the evaluated rows. Insufficient groups are still reported, but they are never the reference group, never fail, and are left out of group extremes and failing counts.
Compute the metric. The selected metric is computed separately for every group and over all evaluated rows.
Choose the comparison point. With reference, each group is compared with the best sufficient group, or with the group you select when using a named reference. With overall, each group is compared with the metric over all evaluated rows. AUC and R² support reference only because their values over all rows are not averages of the groups’ values.
Orient the ratio and gap. When higher is better, the ratio is the group value divided by the comparison value. When lower is better, it is the comparison value divided by the group value. For pass or fail, the gap is measured only on the worse side of the comparison point.
Test significance. With significance_test set to auto, the test uses the statistical test appropriate for the selected metric. You can adjust p-values when comparing many groups and optionally require significance for a group to fail.
Decide which groups fail. A group must be worse than the comparison point. It fails when its ratio is below min_ratio or its gap is greater than max_difference. When require_significance is on, the disparity must also be statistically significant.

Your thresholds are then evaluated against the headline measurements. The test result’s Analysis tab has three sections. Group breakdown shows a summary strip and a Key insights card with facts from the run. Bar graph view charts each group with its confidence interval. Table view lists every group’s numbers and has a Download CSV button. Selecting a group opens records in the Data tab using exact matches on the displayed attribute values and non-empty filters on the measured columns. If the analysis combines equivalent spellings, such as F and F, only the displayed spelling appears in the drill-down. Confidence intervals help you interpret uncertainty but do not affect pass or fail. The AI summary also covers the groups and comparison settings from the run.

Statistical significance

auto tests the quantity that the selected metric measures:

Fisher’s exact test for rate, npv, and false_omission_rate.
Welch’s t-test for mean, mae, log_loss, and brier, and on squared errors for rmse.
A pooled-quantile test for median and p95.
A Hanley-McNeil z-test for auc.
No significance test for r2.

Set significance_test to none when you want the ratio and gap alone to determine the result. Use p_value_adjustment to choose none, holm, or benjamini_hochberg. Verdicts use the adjusted p-value. require_significance is not available for R² because R² has no significance test.

Intersections and small groups

List up to 3 columns in protected_attribute_columns to test every combination of their values, such as gender and age_band. Intersections create more comparisons and can leave groups below min_group_size or min_group_share. Those groups remain visible but are marked insufficient instead of failing. When you test intersections, consider holm or benjamini_hochberg so the extra comparisons do not raise false alarms. A test that exceeds 1,000 groups is skipped, and the result stores up to 100 groups, worst first.

Required columns

Every configuration needs 1 to 3 protected attribute columns. The other required columns depend on parity_metric:

Metric	Required columns and values
`auc`	`score_column`, `ground_truth_column`, and `positive_values`.
`log_loss`, `brier`	`probability_column`, `ground_truth_column`, and `positive_values`.
`npv`, `false_omission_rate`	`prediction_column`, `ground_truth_column`, and `positive_values`.
`mae`, `rmse`, `r2`	Numeric `prediction_column` and `ground_truth_column`.
`mean`, `median`, `p95`	Numeric `value_column` and `higher_is_better`.
`rate`	`value_column`, `positive_values`, and `higher_is_better`.

Column parameters must refer to top-level columns. Fields nested inside a dict or JSON column are not available, so promote them to their own column first. Values in positive_values are matched as text, so ["1"] matches an integer column holding 1, and ["true"] matches a boolean column. An integer written with a decimal point, such as "1.0", matches the same way.

The test is skipped, not failed, when it cannot be computed. This happens when a required column is missing, empty, or not numeric where a number is required; when a selected positive class does not occur where the metric requires one; when fewer than 2 groups are sufficient; when the metric is undefined for every sufficient group; and when the protected attributes form more than 1,000 groups. The skip message says which.

A ratio needs a positive comparison point. If a worse group has an undefined ratio and Max allowed gap is not set, the test skips and tells you to set max_difference. This commonly matters for lower-is-better rates when the reference value is 0.

Parameters

Parameter	Required	Default	Description
`protected_attribute_columns`	Yes		List of 1 to 3 columns that define the groups.
`parity_metric`	Yes		`auc`, `log_loss`, `brier`, `npv`, `false_omission_rate`, `mae`, `rmse`, `r2`, `mean`, `median`, `p95`, or `rate`.
`score_column`	For AUC		Predicted score for the positive class.
`probability_column`	For loss		Predicted probability of the positive class, from 0 to 1. Used by `log_loss` and `brier`.
`prediction_column`	As applicable		Predicted label for `npv` and `false_omission_rate`, or numeric prediction for `mae`, `rmse`, and `r2`.
`ground_truth_column`	Model metrics		Ground-truth label or numeric target.
`value_column`	Column metrics		Column used by `mean`, `median`, `p95`, or `rate`.
`positive_values`	As applicable		Values that count as the positive class, or as positive rows for `rate`.
`higher_is_better`	Column metrics		Direction for `mean`, `median`, `p95`, and `rate`. Required with no default.
`comparison`	No	`reference`	Compare with: `reference` compares with a reference group; `overall` compares with all rows. AUC and R² support `reference` only.
`reference_strategy`	No	`highest`	With `reference`, `highest` selects the best sufficient group for the metric’s direction; `named` uses `reference_value`.
`reference_value`	No		The named reference group’s value. Required with `named`, which needs exactly one protected attribute column.
`min_ratio`	No	`0.8`	Min allowed ratio. A worse group fails below this direction-aware ratio. Greater than 0 and at most 1.
`max_difference`	No		Max allowed gap. A worse group also fails above this non-negative gap in the metric’s units. For a rate metric, the value is at most 1.
`significance_test`	No	`auto`	`auto` selects the test for the metric; `none` disables pairwise significance testing.
`alpha`	No	`0.05`	Significance level, strictly between 0 and 1.
`p_value_adjustment`	No	`none`	`none`, `holm`, or `benjamini_hochberg`.
`require_significance`	No	`false`	When `true`, a group fails only if its disparity is also significant. Not available for R².
`min_group_size`	No	`30`	Minimum rows for a group to be sufficient.
`min_group_share`	No	`0.02`	Minimum share of evaluated rows, from 0 to 1, for a group to be sufficient.

Measurements

Thresholds are set on one of the measurements below.

Measurement	Description
`failingGroupCount`	Number of groups that fail. Use `<= 0` for “no group fails”.
`failingGroupPercentage`	Share of sufficient groups that fail, as a fraction between 0 and 1.
`minMetricRatio`	Lowest direction-aware ratio among compared groups. Typically `>= 0.8`. Empty when a worse group has no defined ratio.
`maxMetricDifference`	Largest shortfall from the comparison point on the worse side, in the metric’s units. `0` when no group is worse.
`worstGroupValue`	Worst value among sufficient groups, including the reference: the lowest when higher is better and the highest when lower is better.
`significantDisparityCount`	Number of groups that are significantly worse than the comparison point.
`insufficientGroupCount`	Number of groups below the minimum size or share.
`missingAttributeRate`	Share of rows with the metric inputs excluded because at least one protected attribute was missing.
`minGroupShare`	Smallest group’s share of the evaluated rows.

Test configuration examples

If you are writing a tests.json, here are a few valid configurations for the group metric parity test:

[
  {
    "name": "AUC parity by gender",
    "description": "Ensures every sufficient gender group's AUC is at least 80% of the best group's AUC",
    "type": "fairness",
    "subtype": "groupMetricParity",
    "thresholds": [
      {
        "insightName": "groupMetricParity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "parity_metric", "value": "auc" },
          { "name": "score_column", "value": "approval_score" },
          { "name": "ground_truth_column", "value": "approved" },
          { "name": "positive_values", "value": ["1"] }
        ],
        "measurement": "minMetricRatio",
        "operator": ">=",
        "value": 0.8
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "a91dba8d-a9e6-4696-9d38-b79d477d629f"
  },
  {
    "name": "Judge score parity by language and region",
    "description": "Fails a language and region intersection only when its mean judge score is below 80% of the best group's and the gap is significant",
    "type": "fairness",
    "subtype": "groupMetricParity",
    "thresholds": [
      {
        "insightName": "groupMetricParity",
        "insightParameters": [
          {
            "name": "protected_attribute_columns",
            "value": ["language", "region"]
          },
          { "name": "parity_metric", "value": "mean" },
          { "name": "value_column", "value": "judge_score" },
          { "name": "higher_is_better", "value": true },
          { "name": "p_value_adjustment", "value": "holm" },
          { "name": "require_significance", "value": true }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "8ff6d6c2-4692-4bdb-95e5-2273cc51f29d"
  }
]

[
  {
    "name": "Weekly p95 latency parity by region",
    "description": "Monitors each region's p95 latency against all rows with a 500 ms maximum gap over a 7-day window",
    "type": "fairness",
    "subtype": "groupMetricParity",
    "thresholds": [
      {
        "insightName": "groupMetricParity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["region"] },
          { "name": "parity_metric", "value": "p95" },
          { "name": "value_column", "value": "latency" },
          { "name": "higher_is_better", "value": false },
          { "name": "comparison", "value": "overall" },
          { "name": "max_difference", "value": 500 }
        ],
        "measurement": "maxMetricDifference",
        "operator": "<=",
        "value": 500
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "6a040f3b-6f8f-4acb-9dcc-5fe05316bf52"
  },
  {
    "name": "Weekly refusal rate parity by language",
    "description": "Monitors whether any language's refusal rate is more than 5 percentage points above the rate across all rows",
    "type": "fairness",
    "subtype": "groupMetricParity",
    "thresholds": [
      {
        "insightName": "groupMetricParity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["language"] },
          { "name": "parity_metric", "value": "rate" },
          { "name": "value_column", "value": "refused" },
          { "name": "positive_values", "value": ["true"] },
          { "name": "higher_is_better", "value": false },
          { "name": "comparison", "value": "overall" },
          { "name": "max_difference", "value": 0.05 }
        ],
        "measurement": "maxMetricDifference",
        "operator": "<=",
        "value": 0.05
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "e4025455-cb5d-49c3-b99d-45841b7b3694"
  }
]

Group outcome disparity test - Compare favorable outcome rates across protected groups.
Group score disparity test - Compare a numeric score across protected groups.
Group performance parity test - Compare error rates and performance metrics across protected groups.
Group by column statistic test - Check a simple statistic for each value of a column.

Get started

Workspace setup

Governance

Observability

Offline testing

Tests

Gateway

Data quality monitoring

Administration

Notifications

Other resources

Group metric parity

Definition

Taxonomy

Why it matters

Metrics by task type

How it works

Statistical significance

Intersections and small groups

Required columns

Parameters

Measurements

Test configuration examples

​Definition

​Taxonomy

​Why it matters

​Metrics by task type

​How it works

​Statistical significance

​Intersections and small groups

​Required columns

​Parameters

​Measurements

​Test configuration examples

​Related

Definition

Taxonomy

Why it matters

Metrics by task type

How it works

Statistical significance

Intersections and small groups

Required columns

Parameters

Measurements

Test configuration examples

Related