Group outcome disparity

Definition

The group outcome disparity test compares how often each protected group (for example, each gender or age band) receives a favorable outcome, such as an approval or a selection. For every group it computes the favorable-outcome rate, the impact ratio against a reference group, and the rate difference, and it can test whether the gap is statistically significant. Each run produces a table with one row per group plus a handful of headline measurements, such as the lowest impact ratio across groups. You set thresholds on those measurements, and the test passes or fails like any other Openlayer test. You can combine up to 3 protected attributes to test intersections, such as gender and age band together.

Taxonomy

Task types: LLM, tabular classification, tabular regression, text classification.
Availability: and .

Pipelines backed by BigQuery or Snowflake are not yet supported by the fairness tests. The test is skipped on them.

Why it matters

Differences in favorable-outcome rates between groups are the starting point of an adverse-impact analysis for decisions about people, such as hiring, lending, insurance, and claims. The impact ratio and the four-fifths rule are the screening numbers most reviewers ask for first.
A raw ratio is noisy for small groups. The significance layer and the small-sample guard keep a tiny group that happens to have a 100% rate from becoming the yardstick, and keep a three-row group from failing the test by chance.
Intersections surface disparities that single attributes hide, such as a group that is disadvantaged only in combination with another attribute.
On a monitoring schedule, the test catches disparities that appear after a model, prompt, or population change, not only at launch.
The result names the exact groups behind a failure, so reviewers see the evidence instead of a single number.

The four-fifths flag is an adverse-impact indicator: a screening heuristic that tells you where to look. It is not a legal determination. A ratio above or below 0.8 is not, on its own, evidence that a model is or is not discriminatory.

How it works

The test follows these steps on the rows of each evaluation window:

Drop incomplete rows. Rows with no outcome are excluded first. Among rows with an outcome, rows with a missing value in any protected attribute column are excluded. A blank or whitespace-only value counts as missing. missingAttributeRate is the share of rows with an outcome that were excluded for a missing attribute.
Form groups. Rows are grouped by the distinct values of the protected attribute columns. With several columns, each combination is its own group, and the group is the same whatever order you list the columns in.
Set small groups aside. A group is sufficient when it has at least min_group_size rows and at least min_group_share of the evaluated rows. Insufficient groups are still listed and counted in insufficientGroupCount, but they are never the reference group, never fail, and are left out of the group extremes and failing counts.
Compute the rate. The rate of a group is its favorable rows divided by its rows.
Choose the reference group. With highest (the default) it is the sufficient group with the highest rate. With named it is the group you set in reference_value.
Compare each group to the reference. The impact ratio is the group rate divided by the reference rate, and the rate difference is the group rate minus the reference rate. Every sufficient group with an impact ratio below 0.8 gets the four-fifths flag, whatever impact_ratio_threshold is.
Test significance. Each group is tested against the reference group, as described in Statistical significance.
Decide which groups fail. A group fails when its impact ratio is below impact_ratio_threshold. When require_significance is on, it must also be a statistically significant disparity.

Your thresholds are then evaluated against the headline measurements. The test result’s Analysis tab has three sections. Group breakdown shows a summary strip and a Key insights card with facts from the run. Bar graph view charts each group with its confidence interval. Table view lists every group’s numbers and has a Download CSV button. Selecting a group opens records in the Data tab using exact matches on the displayed attribute values and non-empty filters on the measured columns. If the analysis combines equivalent spellings, such as F and F, only the displayed spelling appears in the drill-down. Confidence intervals help you interpret uncertainty but do not affect pass or fail.

Statistical significance

The significance_test parameter sets the test that compares each group with the reference group:

Value	Test
`fisher_exact`	Default. Two-sided Fisher’s exact test on the favorable and unfavorable counts. Reliable for small counts.
`z_test`	Two-sided two-proportion z-test with a pooled variance.
`chi_square`	Chi-square test of independence on the same 2x2 table, with Yates continuity correction.
`none`	No pairwise test. Groups are compared on the impact ratio and rate difference only.

Whatever you choose, every run also reports an omnibus chi-square test across all sufficient groups as chiSquarePValue. Comparing many groups at once, especially intersections, makes some false positives likely. Use p_value_adjustment to correct for it:

none (default) uses the raw p-values.
holm applies the Holm step-down correction, which controls the chance of any false positive.
benjamini_hochberg applies the Benjamini-Hochberg correction, which controls the share of false positives among the groups flagged.

Raw and adjusted p-values are both stored and shown, and verdicts use the adjusted value. A group counts toward significantDisparityCount when its adjusted p-value is below alpha and its rate is below the reference rate. A small p-value says a gap is unlikely to be chance. It does not say whether the gap is justified.

Intersections

List up to 3 columns in protected_attribute_columns to test every combination of their values, for example gender and age_band. Intersections split each window into many small groups, so a short evaluation window can leave several of them below min_group_size. Those groups are reported as insufficient rather than judged. If you want them judged, evaluate a longer window or lower min_group_size, and consider holm or benjamini_hochberg so the extra comparisons do not raise false alarms. A test that exceeds 1,000 groups is skipped, since the attribute combination is probably high-cardinality. The result stores up to 100 groups, worst first.

Required columns

To compute this metric, your dataset must contain the following columns:

Protected attributes: 1 to 3 columns holding the attribute values, such as gender or age_band.
Outcome: The column holding the decision or result, such as approved.

Column parameters must refer to top-level columns. Fields nested inside a dict or JSON column are not available, so promote them to their own column first. Values in favorable_values are matched as text, so ["1"] matches an integer column holding 1, and ["true"] matches a boolean column. An integer written with a decimal point, such as "1.0", matches the same way, and the values 1 and 1.0 in a text attribute column form one group. Rows whose outcome matches none of the favorable values count as unfavorable.

The test is skipped, not failed, when it cannot be computed. This happens when a column is missing or empty in the window, when fewer than 2 groups are sufficient, when favorable_values matches no rows, when the reference group has no favorable outcomes, when a named reference group has no rows or falls below the minimum size or share, when the protected attributes form more than 1,000 groups, and when control_columns have 1,000 or more combinations. The skip message says which.

Parameters

Parameter	Required	Default	Description
`protected_attribute_columns`	Yes		List of 1 to 3 columns that define the groups.
`outcome_column`	Yes		Column holding the outcome.
`favorable_values`	Yes		List of outcome values that count as favorable.
`reference_strategy`	No	`highest`	`highest` compares to the highest-rate sufficient group. `named` compares to the group in `reference_value`.
`reference_value`	No		The reference group’s value. Required with `named`, which needs exactly one protected attribute column.
`impact_ratio_threshold`	No	`0.8`	A group fails when its impact ratio is below this value. Greater than 0 and at most 1.
`significance_test`	No	`fisher_exact`	`none`, `z_test`, `fisher_exact`, or `chi_square`.
`alpha`	No	`0.05`	Significance level, strictly between 0 and 1.
`p_value_adjustment`	No	`none`	`none`, `holm`, or `benjamini_hochberg`.
`require_significance`	No	`false`	When `true`, a group fails only if it is also a significant disparity. Needs a `significance_test` other than `none`.
`min_group_size`	No	`30`	Minimum rows for a group to be sufficient.
`min_group_share`	No	`0.02`	Minimum share of evaluated rows, from 0 to 1, for a group to be sufficient.
`control_columns`	No		List of 1 to 2 columns to stratify by for the adjusted impact ratio. Can’t include `outcome_column`. See below.

Measurements

Thresholds are set on one of the measurements below.

Measurement	Description
`failingGroupCount`	Number of groups that fail. Use `<= 0` for “no group fails”.
`failingGroupPercentage`	Share of sufficient groups that fail, as a fraction between 0 and 1.
`minImpactRatio`	Lowest impact ratio among the non-reference sufficient groups. Typically `>= 0.8`.
`maxRateDifference`	Largest shortfall in rate vs the reference group: the biggest gap of a group below it, or `0` when none is. A group ahead of a named reference does not count.
`fourFifthsViolationCount`	Number of groups below the four-fifths (0.8) impact ratio. An indicator count, not a legal violation.
`significantDisparityCount`	Number of groups that are significantly disadvantaged compared with the reference.
`chiSquarePValue`	P-value of the omnibus chi-square test across sufficient groups. Typically `>= 0.05`.
`insufficientGroupCount`	Number of groups below the minimum size or share.
`missingAttributeRate`	Share of rows with an outcome excluded because at least one protected attribute was missing.
`minGroupShare`	Smallest group’s share of the evaluated rows.
`adjustedMinImpactRatio`	Lowest Mantel-Haenszel adjusted impact ratio across groups, after stratifying by `control_columns`. Typically `>= 0.8`. Only available with `control_columns`.
`adjustedMinOddsRatio`	Lowest Mantel-Haenszel adjusted odds ratio across groups. Informational: it is an odds ratio, not an impact ratio, so 0.8 does not apply. Only available with `control_columns`.

With control_columns, each group is also compared with the reference within every combination of the control columns, such as region, and the comparisons are pooled with Mantel-Haenszel weights. This shows whether a gap remains when groups are compared like for like. Combinations where the group or the reference has no rows are left out. The adjusted impact ratio is a pooled ratio of favorable rates, so it reads like the impact ratio and the 0.8 convention applies to it. The adjusted odds ratio is a different quantity that sits further from 1 for the same gap, so do not read it against 0.8. Which groups fail is still decided on the unadjusted impact ratio.

Test configuration examples

If you are writing a tests.json, here are a few valid configurations for the group outcome disparity test:

[
  {
    "name": "Approval rate parity by gender",
    "description": "Ensures no gender group has an approval impact ratio below 0.8 compared with the highest-rate group",
    "type": "fairness",
    "subtype": "groupOutcomeDisparity",
    "thresholds": [
      {
        "insightName": "groupOutcomeDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "outcome_column", "value": "approved" },
          { "name": "favorable_values", "value": ["1"] }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "c0725580-9015-4f24-9f38-7b1875535475"
  },
  {
    "name": "Approval rate parity by gender and age band",
    "description": "Fails a gender and age band intersection only when it is below 0.8 and a significant disparity after Holm correction",
    "type": "fairness",
    "subtype": "groupOutcomeDisparity",
    "thresholds": [
      {
        "insightName": "groupOutcomeDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender", "age_band"] },
          { "name": "outcome_column", "value": "approved" },
          { "name": "favorable_values", "value": ["1"] },
          { "name": "significance_test", "value": "fisher_exact" },
          { "name": "p_value_adjustment", "value": "holm" },
          { "name": "require_significance", "value": true },
          { "name": "min_group_size", "value": 30 }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "0a8a9459-d347-4b16-9394-1fa6fd9402e5"
  }
]

[
  {
    "name": "Weekly minimum impact ratio by gender",
    "description": "Monitors that the lowest gender impact ratio stays at or above 0.8 over a 7-day window",
    "type": "fairness",
    "subtype": "groupOutcomeDisparity",
    "thresholds": [
      {
        "insightName": "groupOutcomeDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "outcome_column", "value": "approved" },
          { "name": "favorable_values", "value": ["1"] }
        ],
        "measurement": "minImpactRatio",
        "operator": ">=",
        "value": 0.8
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "e7cb3d1f-5534-468c-a28a-2a6e08dc5d46"
  },
  {
    "name": "Weekly approval rate parity by gender and age band",
    "description": "Fails a gender and age band intersection only when its impact ratio is below 0.8 and the disparity is significant after Holm correction, over a 7-day window",
    "type": "fairness",
    "subtype": "groupOutcomeDisparity",
    "thresholds": [
      {
        "insightName": "groupOutcomeDisparity",
        "insightParameters": [
          {
            "name": "protected_attribute_columns",
            "value": ["gender", "age_band"]
          },
          { "name": "outcome_column", "value": "approved" },
          { "name": "favorable_values", "value": ["1"] },
          { "name": "p_value_adjustment", "value": "holm" },
          { "name": "require_significance", "value": true },
          { "name": "min_group_size", "value": 20 }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "7e62c4a2-6063-421b-b057-75fe876c1419"
  }
]

Group score disparity test - Compare a numeric score across protected groups.
Group performance parity test - Compare error rates and performance metrics across protected groups.
Group metric parity test - Compare model metrics and column statistics across protected groups.
Group by column statistic test - Check a simple statistic for each value of a column.
Bias test - Detect bias in LLM outputs with an LLM-as-a-judge.

Get started

Workspace setup

Governance

Observability

Offline testing

Tests

Gateway

Data quality monitoring

Administration

Notifications

Other resources

Group outcome disparity

Definition

Taxonomy

Why it matters

How it works

Statistical significance

Intersections

Required columns

Parameters

Measurements

Test configuration examples

​Definition

​Taxonomy

​Why it matters

​How it works

​Statistical significance

​Intersections

​Required columns

​Parameters

​Measurements

​Test configuration examples

​Related

Definition

Taxonomy

Why it matters

How it works

Statistical significance

Intersections

Required columns

Parameters

Measurements

Test configuration examples

Related