> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Group outcome disparity

> Learn how to use the group outcome disparity test to compare favorable outcome rates, impact ratios, and statistical significance across protected groups

## Definition

The group outcome disparity test compares how often each protected group (for
example, each gender or age band) receives a favorable outcome, such as an
approval or a selection. For every group it computes the favorable-outcome rate,
the **impact ratio** against a reference group, and the **rate difference**, and
it can test whether the gap is statistically significant.

Each run produces a table with one row per group plus a handful of headline
measurements, such as the lowest impact ratio across groups. You set thresholds
on those measurements, and the test passes or fails like any other Openlayer
test. You can combine up to 3 protected attributes to test **intersections**,
such as gender and age band together.

## Taxonomy

* **Task types**: LLM, tabular classification, tabular regression, text classification.
* **Availability**: development (Continuously evaluate your models and datasets as you iterate on their versions.)
  and monitoring (Monitor a model in production, measure its health, check for drifts and set up alerts.).

> [!NOTE]
>
> Pipelines backed by BigQuery or Snowflake are not yet supported by the
> fairness tests. The test is skipped on them.

## Why it matters

* Differences in favorable-outcome rates between groups are the starting point
  of an adverse-impact analysis for decisions about people, such as hiring,
  lending, insurance, and claims. The impact ratio and the four-fifths rule are
  the screening numbers most reviewers ask for first.
* A raw ratio is noisy for small groups. The significance layer and the
  small-sample guard keep a tiny group that happens to have a 100% rate from
  becoming the yardstick, and keep a three-row group from failing the test by
  chance.
* Intersections surface disparities that single attributes hide, such as a
  group that is disadvantaged only in combination with another attribute.
* On a monitoring schedule, the test catches disparities that appear after a
  model, prompt, or population change, not only at launch.
* The result names the exact groups behind a failure, so reviewers see the
  evidence instead of a single number.

> [!NOTE]
>
> The four-fifths flag is an **adverse-impact indicator**: a screening heuristic
> that tells you where to look. It is not a legal determination. A ratio above
> or below 0.8 is not, on its own, evidence that a model is or is not
> discriminatory.

## How it works

The test follows these steps on the rows of each evaluation window:

1. **Drop incomplete rows.** Rows with no outcome are excluded first. Among
   rows with an outcome, rows with a missing value in any protected attribute
   column are excluded. A blank or whitespace-only value counts as missing.
   `missingAttributeRate` is the share of rows with an outcome that were
   excluded for a missing attribute.
2. **Form groups.** Rows are grouped by the distinct values of the protected
   attribute columns. With several columns, each combination is its own group,
   and the group is the same whatever order you list the columns in.
3. **Set small groups aside.** A group is *sufficient* when it has at least
   `min_group_size` rows and at least `min_group_share` of the evaluated rows.
   Insufficient groups are still listed and counted in `insufficientGroupCount`,
   but they are never the reference group, never fail, and are left out of the
   group extremes and failing counts.
4. **Compute the rate.** The rate of a group is its favorable rows divided by
   its rows.
5. **Choose the reference group.** With `highest` (the default) it is the
   sufficient group with the highest rate. With `named` it is the group you set
   in `reference_value`.
6. **Compare each group to the reference.** The impact ratio is the group rate
   divided by the reference rate, and the rate difference is the group rate
   minus the reference rate. Every sufficient group with an impact ratio below
   0.8 gets the four-fifths flag, whatever `impact_ratio_threshold` is.
7. **Test significance.** Each group is tested against the reference group, as
   described in [Statistical significance](#statistical-significance).
8. **Decide which groups fail.** A group fails when its impact ratio is below
   `impact_ratio_threshold`. When `require_significance` is on, it must also be
   a statistically significant disparity.

Your thresholds are then evaluated against the headline measurements. The test
result's **Analysis** tab has three sections. **Group breakdown** shows a
summary strip and a **Key insights** card with facts from the run. **Bar graph
view** charts each group with its confidence interval. **Table view** lists
every group's numbers and has a **Download CSV** button. Selecting a group opens
records in the **Data** tab using exact matches on the displayed attribute
values and non-empty filters on the measured columns. If the analysis combines
equivalent spellings, such as `F` and ` F`, only the displayed spelling appears
in the drill-down. Confidence intervals help you interpret uncertainty but do
not affect pass or fail.

## Statistical significance

The `significance_test` parameter sets the test that compares each group with
the reference group:

| Value | Test |
| - | - |
| `fisher_exact` | Default. Two-sided Fisher's exact test on the favorable and unfavorable counts. Reliable for small counts. |
| `z_test` | Two-sided two-proportion z-test with a pooled variance. |
| `chi_square` | Chi-square test of independence on the same 2x2 table, with Yates continuity correction. |
| `none` | No pairwise test. Groups are compared on the impact ratio and rate difference only. |

Whatever you choose, every run also reports an omnibus chi-square test across
all sufficient groups as `chiSquarePValue`.

Comparing many groups at once, especially intersections, makes some false
positives likely. Use `p_value_adjustment` to correct for it:

* `none` (default) uses the raw p-values.
* `holm` applies the Holm step-down correction, which controls the chance of any
  false positive.
* `benjamini_hochberg` applies the Benjamini-Hochberg correction, which controls
  the share of false positives among the groups flagged.

Raw and adjusted p-values are both stored and shown, and verdicts use the
adjusted value. A group counts toward `significantDisparityCount` when its
adjusted p-value is below `alpha` and its rate is below the reference rate.

A small p-value says a gap is unlikely to be chance. It does not say whether the
gap is justified.

## Intersections

List up to 3 columns in `protected_attribute_columns` to test every combination
of their values, for example `gender` and `age_band`. Intersections split each
window into many small groups, so a short evaluation window can leave several of
them below `min_group_size`. Those groups are reported as insufficient rather
than judged. If you want them judged, evaluate a longer window or lower
`min_group_size`, and consider `holm` or `benjamini_hochberg` so the extra
comparisons do not raise false alarms.

A test that exceeds 1,000 groups is skipped, since the attribute combination is
probably high-cardinality. The result stores up to 100 groups, worst first.

## Required columns

To compute this metric, your dataset must contain the following columns:

* **Protected attributes**: 1 to 3 columns holding the attribute values, such as `gender` or `age_band`.
* **Outcome**: The column holding the decision or result, such as `approved`.

Column parameters must refer to top-level columns. Fields nested inside a dict
or JSON column are not available, so promote them to their own column first.

Values in `favorable_values` are matched as text, so `["1"]` matches an integer
column holding `1`, and `["true"]` matches a boolean column. An integer written
with a decimal point, such as `"1.0"`, matches the same way, and the values `1`
and `1.0` in a text attribute column form one group. Rows whose outcome matches
none of the favorable values count as unfavorable.

> [!NOTE]
>
> The test is skipped, not failed, when it cannot be computed. This happens when
> a column is missing or empty in the window, when fewer than 2 groups are
> sufficient, when `favorable_values` matches no rows, when the reference group
> has no favorable outcomes, when a `named` reference group has no rows or falls
> below the minimum size or share, when the protected attributes form more than
> 1,000 groups, and when `control_columns` have 1,000 or more combinations. The
> skip message says which.

## Parameters

| Parameter | Required | Default | Description |
| - | - | - | - |
| `protected_attribute_columns` | Yes | | List of 1 to 3 columns that define the groups. |
| `outcome_column` | Yes | | Column holding the outcome. |
| `favorable_values` | Yes | | List of outcome values that count as favorable. |
| `reference_strategy` | No | `highest` | `highest` compares to the highest-rate sufficient group. `named` compares to the group in `reference_value`. |
| `reference_value` | No | | The reference group's value. Required with `named`, which needs exactly one protected attribute column. |
| `impact_ratio_threshold` | No | `0.8` | A group fails when its impact ratio is below this value. Greater than 0 and at most 1. |
| `significance_test` | No | `fisher_exact` | `none`, `z_test`, `fisher_exact`, or `chi_square`. |
| `alpha` | No | `0.05` | Significance level, strictly between 0 and 1. |
| `p_value_adjustment` | No | `none` | `none`, `holm`, or `benjamini_hochberg`. |
| `require_significance` | No | `false` | When `true`, a group fails only if it is also a significant disparity. Needs a `significance_test` other than `none`. |
| `min_group_size` | No | `30` | Minimum rows for a group to be sufficient. |
| `min_group_share` | No | `0.02` | Minimum share of evaluated rows, from 0 to 1, for a group to be sufficient. |
| `control_columns` | No | | List of 1 to 2 columns to stratify by for the adjusted impact ratio. Can't include `outcome_column`. See below. |

## Measurements

Thresholds are set on one of the measurements below.

| Measurement | Description |
| - | - |
| `failingGroupCount` | Number of groups that fail. Use `<= 0` for "no group fails". |
| `failingGroupPercentage` | Share of sufficient groups that fail, as a fraction between 0 and 1. |
| `minImpactRatio` | Lowest impact ratio among the non-reference sufficient groups. Typically `>= 0.8`. |
| `maxRateDifference` | Largest shortfall in rate vs the reference group: the biggest gap of a group below it, or `0` when none is. A group ahead of a named reference does not count. |
| `fourFifthsViolationCount` | Number of groups below the four-fifths (0.8) impact ratio. An indicator count, not a legal violation. |
| `significantDisparityCount` | Number of groups that are significantly disadvantaged compared with the reference. |
| `chiSquarePValue` | P-value of the omnibus chi-square test across sufficient groups. Typically `>= 0.05`. |
| `insufficientGroupCount` | Number of groups below the minimum size or share. |
| `missingAttributeRate` | Share of rows with an outcome excluded because at least one protected attribute was missing. |
| `minGroupShare` | Smallest group's share of the evaluated rows. |
| `adjustedMinImpactRatio` | Lowest Mantel-Haenszel adjusted impact ratio across groups, after stratifying by `control_columns`. Typically `>= 0.8`. Only available with `control_columns`. |
| `adjustedMinOddsRatio` | Lowest Mantel-Haenszel adjusted odds ratio across groups. Informational: it is an odds ratio, not an impact ratio, so 0.8 does not apply. Only available with `control_columns`. |

With `control_columns`, each group is also compared with the reference within
every combination of the control columns, such as `region`, and the comparisons
are pooled with Mantel-Haenszel weights. This shows whether a gap remains when
groups are compared like for like. Combinations where the group or the
reference has no rows are left out. The adjusted impact ratio is a pooled ratio
of favorable rates, so it reads like the impact ratio and the 0.8 convention
applies to it. The adjusted odds ratio is a different quantity that sits
further from 1 for the same gap, so do not read it against 0.8. Which groups
fail is still decided on the unadjusted impact ratio.

## Test configuration examples

If you are writing a `tests.json`, here are a few valid configurations for the group outcome disparity test:

```json Development theme={null}
[
  {
    "name": "Approval rate parity by gender",
    "description": "Ensures no gender group has an approval impact ratio below 0.8 compared with the highest-rate group",
    "type": "fairness",
    "subtype": "groupOutcomeDisparity",
    "thresholds": [
      {
        "insightName": "groupOutcomeDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "outcome_column", "value": "approved" },
          { "name": "favorable_values", "value": ["1"] }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "c0725580-9015-4f24-9f38-7b1875535475"
  },
  {
    "name": "Approval rate parity by gender and age band",
    "description": "Fails a gender and age band intersection only when it is below 0.8 and a significant disparity after Holm correction",
    "type": "fairness",
    "subtype": "groupOutcomeDisparity",
    "thresholds": [
      {
        "insightName": "groupOutcomeDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender", "age_band"] },
          { "name": "outcome_column", "value": "approved" },
          { "name": "favorable_values", "value": ["1"] },
          { "name": "significance_test", "value": "fisher_exact" },
          { "name": "p_value_adjustment", "value": "holm" },
          { "name": "require_significance", "value": true },
          { "name": "min_group_size", "value": 30 }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "0a8a9459-d347-4b16-9394-1fa6fd9402e5"
  }
]
```

```json Monitoring theme={null}
[
  {
    "name": "Weekly minimum impact ratio by gender",
    "description": "Monitors that the lowest gender impact ratio stays at or above 0.8 over a 7-day window",
    "type": "fairness",
    "subtype": "groupOutcomeDisparity",
    "thresholds": [
      {
        "insightName": "groupOutcomeDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "outcome_column", "value": "approved" },
          { "name": "favorable_values", "value": ["1"] }
        ],
        "measurement": "minImpactRatio",
        "operator": ">=",
        "value": 0.8
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "e7cb3d1f-5534-468c-a28a-2a6e08dc5d46"
  },
  {
    "name": "Weekly approval rate parity by gender and age band",
    "description": "Fails a gender and age band intersection only when its impact ratio is below 0.8 and the disparity is significant after Holm correction, over a 7-day window",
    "type": "fairness",
    "subtype": "groupOutcomeDisparity",
    "thresholds": [
      {
        "insightName": "groupOutcomeDisparity",
        "insightParameters": [
          {
            "name": "protected_attribute_columns",
            "value": ["gender", "age_band"]
          },
          { "name": "outcome_column", "value": "approved" },
          { "name": "favorable_values", "value": ["1"] },
          { "name": "p_value_adjustment", "value": "holm" },
          { "name": "require_significance", "value": true },
          { "name": "min_group_size", "value": 20 }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "7e62c4a2-6063-421b-b057-75fe876c1419"
  }
]
```

## Related

* [Group score disparity test](/tests/catalog/group-score-disparity) - Compare a numeric score across protected groups.
* [Group performance parity test](/tests/catalog/group-performance-parity) - Compare error rates and performance metrics across protected groups.
* [Group metric parity test](/tests/catalog/group-metric-parity) - Compare model metrics and column statistics across protected groups.
* [Group by column statistic test](/tests/catalog/group-by-column-statistic) - Check a simple statistic for each value of a column.
* [Bias test](/tests/catalog/bias) - Detect bias in LLM outputs with an LLM-as-a-judge.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.