> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Group metric parity

> Learn how to use the group metric parity test to compare model metrics and column statistics across protected groups

## Definition

The group metric parity test compares one metric across protected groups, such
as each gender or age band. You can compare model-quality metrics such as AUC,
log loss, MAE, or R², or the mean, median, p95, or rate of a column such as a
judge score, latency, cost, refusal flag, or tool-error flag.

For every group, the test computes the metric's value, its direction-aware ratio
to a comparison point, and its gap from that point. A ratio below 1 always means
that the group is worse, whether higher or lower values are better. You can
compare each group with a reference group or with the metric over all rows, and
you can combine up to 3 protected attributes to test **intersections**.

## Taxonomy

* **Task types**: LLM, tabular classification, tabular regression, text classification.
* **Availability**: development (Continuously evaluate your models and datasets as you iterate on their versions.)
  and monitoring (Monitor a model in production, measure its health, check for drifts and set up alerts.).

> [!NOTE]
>
> Pipelines backed by BigQuery or Snowflake are not yet supported by the
> fairness tests. The test is skipped on them.

## Why it matters

* Aggregate quality can hide a model that performs well overall but has a much
  higher error or loss for one group.
* The column metrics extend parity checks beyond model accuracy. For example,
  you can compare judge scores, latency, cost, refusals, or tool errors to find
  a quality-of-service gap that affects only one group.
* The ratio gives you a scale-independent screen, while the gap shows the
  shortfall in the metric's own units.
* The small-group guard and significance options help you distinguish a
  repeatable disparity from noise in a short evaluation window.
* On a monitoring schedule, the test catches group-level changes before they
  are visible in the overall metric.

## Metrics by task type

The available metrics depend on the project's task type:

| Task type | Available metrics |
| - | - |
| LLM | AUC, log loss, Brier score, NPV, false omission rate, MAE, RMSE, R², mean, median, p95, and rate. |
| Tabular classification and text classification | AUC, log loss, Brier score, NPV, false omission rate, mean, median, p95, and rate. |
| Tabular regression | MAE, RMSE, R², mean, median, p95, and rate. |

The classification metrics use a ground-truth column and the positive values
you select. AUC reads a score column; log loss and Brier score read a predicted
probability; NPV and false omission rate read a prediction column. MAE, RMSE,
and R² read numeric prediction and ground-truth columns.

The `mean`, `median`, `p95`, and `rate` options can use any suitable top-level
column. For `rate`, you also select the values that count as positive. For all
four column metrics, **Direction** is required and has no default: choose whether
a higher or a lower value is better.

## How it works

The test follows these steps on the rows of each evaluation window:

1. **Drop incomplete rows.** Rows missing an input needed by the selected metric
   are excluded first. Among rows with the metric inputs, rows with a missing
   value in any protected attribute column are excluded. A blank or
   whitespace-only attribute counts as missing. `missingAttributeRate` is the
   share of rows with the metric inputs that were excluded for a missing
   attribute.
2. **Form groups and set small ones aside.** Rows are grouped by the distinct
   values of the protected attribute columns. A group is *sufficient* when it
   has at least `min_group_size` rows and at least `min_group_share` of the
   evaluated rows. Insufficient groups are still reported, but they are never
   the reference group, never fail, and are left out of group extremes and
   failing counts.
3. **Compute the metric.** The selected metric is computed separately for every
   group and over all evaluated rows.
4. **Choose the comparison point.** With `reference`, each group is compared
   with the best sufficient group, or with the group you select when using a
   named reference. With `overall`, each group is compared with the metric over
   all evaluated rows. AUC and R² support `reference` only because their values
   over all rows are not averages of the groups' values.
5. **Orient the ratio and gap.** When higher is better, the ratio is the group
   value divided by the comparison value. When lower is better, it is the
   comparison value divided by the group value. For pass or fail, the gap is
   measured only on the worse side of the comparison point.
6. **Test significance.** With `significance_test` set to `auto`, the test uses
   the statistical test appropriate for the selected metric. You can adjust
   p-values when comparing many groups and optionally require significance for
   a group to fail.
7. **Decide which groups fail.** A group must be worse than the comparison
   point. It fails when its ratio is below `min_ratio` or its gap is greater than
   `max_difference`. When `require_significance` is on, the disparity must also
   be statistically significant.

Your thresholds are then evaluated against the headline measurements. The test
result's **Analysis** tab has three sections. **Group breakdown** shows a
summary strip and a **Key insights** card with facts from the run. **Bar graph
view** charts each group with its confidence interval. **Table view** lists
every group's numbers and has a **Download CSV** button. Selecting a group opens
records in the **Data** tab using exact matches on the displayed attribute
values and non-empty filters on the measured columns. If the analysis combines
equivalent spellings, such as `F` and ` F`, only the displayed spelling appears
in the drill-down. Confidence intervals help you interpret uncertainty but do
not affect pass or fail.
The AI summary also covers the groups and comparison settings from the run.

## Statistical significance

`auto` tests the quantity that the selected metric measures:

* Fisher's exact test for `rate`, `npv`, and `false_omission_rate`.
* Welch's t-test for `mean`, `mae`, `log_loss`, and `brier`, and on squared
  errors for `rmse`.
* A pooled-quantile test for `median` and `p95`.
* A Hanley-McNeil z-test for `auc`.
* No significance test for `r2`.

Set `significance_test` to `none` when you want the ratio and gap alone to
determine the result. Use `p_value_adjustment` to choose `none`, `holm`, or
`benjamini_hochberg`. Verdicts use the adjusted p-value. `require_significance`
is not available for R² because R² has no significance test.

## Intersections and small groups

List up to 3 columns in `protected_attribute_columns` to test every combination
of their values, such as `gender` and `age_band`. Intersections create more
comparisons and can leave groups below `min_group_size` or `min_group_share`.
Those groups remain visible but are marked insufficient instead of failing.

When you test intersections, consider `holm` or `benjamini_hochberg` so the
extra comparisons do not raise false alarms. A test that exceeds 1,000 groups
is skipped, and the result stores up to 100 groups, worst first.

## Required columns

Every configuration needs 1 to 3 **protected attribute** columns. The other
required columns depend on `parity_metric`:

| Metric | Required columns and values |
| - | - |
| `auc` | `score_column`, `ground_truth_column`, and `positive_values`. |
| `log_loss`, `brier` | `probability_column`, `ground_truth_column`, and `positive_values`. |
| `npv`, `false_omission_rate` | `prediction_column`, `ground_truth_column`, and `positive_values`. |
| `mae`, `rmse`, `r2` | Numeric `prediction_column` and `ground_truth_column`. |
| `mean`, `median`, `p95` | Numeric `value_column` and `higher_is_better`. |
| `rate` | `value_column`, `positive_values`, and `higher_is_better`. |

Column parameters must refer to top-level columns. Fields nested inside a dict
or JSON column are not available, so promote them to their own column first.

Values in `positive_values` are matched as text, so `["1"]` matches an integer
column holding `1`, and `["true"]` matches a boolean column. An integer written
with a decimal point, such as `"1.0"`, matches the same way.

> [!NOTE]
>
> The test is skipped, not failed, when it cannot be computed. This happens when
> a required column is missing, empty, or not numeric where a number is
> required; when a selected positive class does not occur where the metric
> requires one; when fewer than 2 groups are sufficient; when the metric is
> undefined for every sufficient group; and when the protected attributes form
> more than 1,000 groups. The skip message says which.

> [!NOTE]
>
> A ratio needs a positive comparison point. If a worse group has an undefined
> ratio and **Max allowed gap** is not set, the test skips and tells you to set
> `max_difference`. This commonly matters for lower-is-better rates when the
> reference value is `0`.

## Parameters

| Parameter | Required | Default | Description |
| - | - | - | - |
| `protected_attribute_columns` | Yes | | List of 1 to 3 columns that define the groups. |
| `parity_metric` | Yes | | `auc`, `log_loss`, `brier`, `npv`, `false_omission_rate`, `mae`, `rmse`, `r2`, `mean`, `median`, `p95`, or `rate`. |
| `score_column` | For AUC | | Predicted score for the positive class. |
| `probability_column` | For loss | | Predicted probability of the positive class, from 0 to 1. Used by `log_loss` and `brier`. |
| `prediction_column` | As applicable | | Predicted label for `npv` and `false_omission_rate`, or numeric prediction for `mae`, `rmse`, and `r2`. |
| `ground_truth_column` | Model metrics | | Ground-truth label or numeric target. |
| `value_column` | Column metrics | | Column used by `mean`, `median`, `p95`, or `rate`. |
| `positive_values` | As applicable | | Values that count as the positive class, or as positive rows for `rate`. |
| `higher_is_better` | Column metrics | | **Direction** for `mean`, `median`, `p95`, and `rate`. Required with no default. |
| `comparison` | No | `reference` | **Compare with**: `reference` compares with a reference group; `overall` compares with all rows. AUC and R² support `reference` only. |
| `reference_strategy` | No | `highest` | With `reference`, `highest` selects the best sufficient group for the metric's direction; `named` uses `reference_value`. |
| `reference_value` | No | | The named reference group's value. Required with `named`, which needs exactly one protected attribute column. |
| `min_ratio` | No | `0.8` | **Min allowed ratio**. A worse group fails below this direction-aware ratio. Greater than 0 and at most 1. |
| `max_difference` | No | | **Max allowed gap**. A worse group also fails above this non-negative gap in the metric's units. For a rate metric, the value is at most 1. |
| `significance_test` | No | `auto` | `auto` selects the test for the metric; `none` disables pairwise significance testing. |
| `alpha` | No | `0.05` | Significance level, strictly between 0 and 1. |
| `p_value_adjustment` | No | `none` | `none`, `holm`, or `benjamini_hochberg`. |
| `require_significance` | No | `false` | When `true`, a group fails only if its disparity is also significant. Not available for R². |
| `min_group_size` | No | `30` | Minimum rows for a group to be sufficient. |
| `min_group_share` | No | `0.02` | Minimum share of evaluated rows, from 0 to 1, for a group to be sufficient. |

## Measurements

Thresholds are set on one of the measurements below.

| Measurement | Description |
| - | - |
| `failingGroupCount` | Number of groups that fail. Use `<= 0` for "no group fails". |
| `failingGroupPercentage` | Share of sufficient groups that fail, as a fraction between 0 and 1. |
| `minMetricRatio` | Lowest direction-aware ratio among compared groups. Typically `>= 0.8`. Empty when a worse group has no defined ratio. |
| `maxMetricDifference` | Largest shortfall from the comparison point on the worse side, in the metric's units. `0` when no group is worse. |
| `worstGroupValue` | Worst value among sufficient groups, including the reference: the lowest when higher is better and the highest when lower is better. |
| `significantDisparityCount` | Number of groups that are significantly worse than the comparison point. |
| `insufficientGroupCount` | Number of groups below the minimum size or share. |
| `missingAttributeRate` | Share of rows with the metric inputs excluded because at least one protected attribute was missing. |
| `minGroupShare` | Smallest group's share of the evaluated rows. |

## Test configuration examples

If you are writing a `tests.json`, here are a few valid configurations for the group metric parity test:

```json Development theme={null}
[
  {
    "name": "AUC parity by gender",
    "description": "Ensures every sufficient gender group's AUC is at least 80% of the best group's AUC",
    "type": "fairness",
    "subtype": "groupMetricParity",
    "thresholds": [
      {
        "insightName": "groupMetricParity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "parity_metric", "value": "auc" },
          { "name": "score_column", "value": "approval_score" },
          { "name": "ground_truth_column", "value": "approved" },
          { "name": "positive_values", "value": ["1"] }
        ],
        "measurement": "minMetricRatio",
        "operator": ">=",
        "value": 0.8
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "a91dba8d-a9e6-4696-9d38-b79d477d629f"
  },
  {
    "name": "Judge score parity by language and region",
    "description": "Fails a language and region intersection only when its mean judge score is below 80% of the best group's and the gap is significant",
    "type": "fairness",
    "subtype": "groupMetricParity",
    "thresholds": [
      {
        "insightName": "groupMetricParity",
        "insightParameters": [
          {
            "name": "protected_attribute_columns",
            "value": ["language", "region"]
          },
          { "name": "parity_metric", "value": "mean" },
          { "name": "value_column", "value": "judge_score" },
          { "name": "higher_is_better", "value": true },
          { "name": "p_value_adjustment", "value": "holm" },
          { "name": "require_significance", "value": true }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "8ff6d6c2-4692-4bdb-95e5-2273cc51f29d"
  }
]
```

```json Monitoring theme={null}
[
  {
    "name": "Weekly p95 latency parity by region",
    "description": "Monitors each region's p95 latency against all rows with a 500 ms maximum gap over a 7-day window",
    "type": "fairness",
    "subtype": "groupMetricParity",
    "thresholds": [
      {
        "insightName": "groupMetricParity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["region"] },
          { "name": "parity_metric", "value": "p95" },
          { "name": "value_column", "value": "latency" },
          { "name": "higher_is_better", "value": false },
          { "name": "comparison", "value": "overall" },
          { "name": "max_difference", "value": 500 }
        ],
        "measurement": "maxMetricDifference",
        "operator": "<=",
        "value": 500
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "6a040f3b-6f8f-4acb-9dcc-5fe05316bf52"
  },
  {
    "name": "Weekly refusal rate parity by language",
    "description": "Monitors whether any language's refusal rate is more than 5 percentage points above the rate across all rows",
    "type": "fairness",
    "subtype": "groupMetricParity",
    "thresholds": [
      {
        "insightName": "groupMetricParity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["language"] },
          { "name": "parity_metric", "value": "rate" },
          { "name": "value_column", "value": "refused" },
          { "name": "positive_values", "value": ["true"] },
          { "name": "higher_is_better", "value": false },
          { "name": "comparison", "value": "overall" },
          { "name": "max_difference", "value": 0.05 }
        ],
        "measurement": "maxMetricDifference",
        "operator": "<=",
        "value": 0.05
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "e4025455-cb5d-49c3-b99d-45841b7b3694"
  }
]
```

## Related

* [Group outcome disparity test](/tests/catalog/group-outcome-disparity) - Compare favorable outcome rates across protected groups.
* [Group score disparity test](/tests/catalog/group-score-disparity) - Compare a numeric score across protected groups.
* [Group performance parity test](/tests/catalog/group-performance-parity) - Compare error rates and performance metrics across protected groups.
* [Group by column statistic test](/tests/catalog/group-by-column-statistic) - Check a simple statistic for each value of a column.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.