> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Group score disparity

> Learn how to use the group score disparity test to compare a numeric score, such as a risk score, across protected groups

## Definition

The group score disparity test compares a numeric score, such as a risk score or
a ranking score, across protected groups (for example, each gender or age band).
For every group it computes the mean, median, and **scoring rate**, which is the
share of the group's rows scoring above the overall median. It then compares each
group with a reference group using the scoring-rate **impact ratio**, the mean
and median gaps, and a standardized mean difference, and it can test whether the
gap is statistically significant.

Each run produces a table with one row per group plus a handful of headline
measurements. You set thresholds on those measurements, and the test passes or
fails like any other Openlayer test. You can combine up to 3 protected
attributes to test **intersections**.

## Taxonomy

* **Task types**: LLM, tabular classification, tabular regression, text classification.
* **Availability**: development (Continuously evaluate your models and datasets as you iterate on their versions.)
  and monitoring (Monitor a model in production, measure its health, check for drifts and set up alerts.).

> [!NOTE]
>
> Pipelines backed by BigQuery or Snowflake are not yet supported by the
> fairness tests. The test is skipped on them.

## Why it matters

* Scores often drive decisions indirectly, by ranking candidates, setting a
  price, or deciding who gets a review. A gap in scores between groups can
  produce unequal outcomes even when no single decision looks unfair.
* The scoring rate, the share of a group scoring above the overall median,
  follows the definition used in bias audits of automated employment decision
  tools, such as those under NYC Local Law 144. It turns a continuous score into
  a ratio that can be screened with the same four-fifths rule as a selection
  rate. Using this test does not by itself constitute a bias audit.
* The mean, median, and standardized mean difference show how large the gap is
  in the score's own units, which a ratio alone can hide.
* On a monitoring schedule, the test catches score drift that affects some
  groups more than others.

> [!NOTE]
>
> The four-fifths flag is an **adverse-impact indicator**: a screening heuristic
> that tells you where to look. It is not a legal determination. A ratio above
> or below 0.8 is not, on its own, evidence that a model is or is not
> discriminatory.

## How it works

The test follows these steps on the rows of each evaluation window:

1. **Drop incomplete rows.** Rows with no usable finite score are excluded
   first. Among rows with a usable score, rows with a missing value in any
   protected attribute column are excluded. A blank or whitespace-only value
   counts as missing. `missingAttributeRate` is the share of rows with a usable
   score that were excluded for a missing attribute.
2. **Form groups and set small ones aside.** Rows are grouped by the distinct
   values of the protected attribute columns. A group is *sufficient* when it has
   at least `min_group_size` rows and at least `min_group_share` of the evaluated
   rows. Insufficient groups are listed but never the reference group, never
   fail, and are left out of the group extremes and failing counts. See
   [Intersections](/tests/catalog/group-outcome-disparity#intersections) for how
   this plays out with several attributes.
3. **Compute the scoring rate.** The overall median is taken across all evaluated
   rows. The scoring rate of a group is the share of its rows with a score above
   that median, or below it when `higher_is_favorable` is `false`. Rows exactly
   at the median do not count.
4. **Choose the reference group.** With `highest` (the default) it is the
   sufficient group with the highest scoring rate. With `named` it is the group
   you set in `reference_value`.
5. **Compare each group to the reference.** The scoring-rate impact ratio is the
   group's scoring rate divided by the reference's. The mean and median
   differences are relative to the reference group, and the standardized mean
   difference is the mean difference divided by the pooled standard deviation of
   the two groups. Every sufficient group with a scoring-rate impact ratio below
   0.8 gets the four-fifths flag, whatever `impact_ratio_threshold` is.
6. **Test significance.** Each group is compared with the reference group on the
   share of rows scoring above the overall median, the same share the impact
   ratio is built on. This is a median test: it judges that share, not the whole
   score distribution. The `significance_test` parameter picks the test, and
   `fisher_exact` is the default. Use `p_value_adjustment` to correct for
   comparing many groups. See
   [Statistical significance](/tests/catalog/group-outcome-disparity#statistical-significance)
   for the tests and corrections.
7. **Decide which groups fail.** A group fails when its scoring-rate impact ratio
   is below `impact_ratio_threshold`. When `require_significance` is on, it must
   also be a statistically significant disparity, meaning its adjusted p-value is
   below `alpha` and its scoring rate is below the reference's.

Your thresholds are then evaluated against the headline measurements. The test
result's **Analysis** tab has three sections. **Group breakdown** shows a
summary strip and a **Key insights** card with facts from the run. **Bar graph
view** charts each group with its confidence interval. **Table view** lists
every group's numbers and has a **Download CSV** button. Selecting a group opens
records in the **Data** tab using exact matches on the displayed attribute
values and non-empty filters on the measured columns. If the analysis combines
equivalent spellings, such as `F` and ` F`, only the displayed spelling appears
in the drill-down. Confidence intervals help you interpret uncertainty but do
not affect pass or fail.

> [!NOTE]
>
> The test is skipped, not failed, when it cannot be computed. This happens when
> a column is missing or empty in the window, when the score column is not
> numeric or has only infinite values, when fewer than 2 groups are sufficient,
> when the reference group has no scores beyond the median, when a `named`
> reference group has no rows or falls below the minimum size or share, and when
> the protected attributes form more than 1,000 groups. A reference group has no
> scores beyond the median when scores are heavily tied, such as a 0/1 score
> where most rows are 1, or most rows are 0 when `higher_is_favorable` is
> `false`. For that kind of score, use the [Group outcome disparity
> test](/tests/catalog/group-outcome-disparity) instead. The skip message says
> which.

## Required columns

To compute this metric, your dataset must contain the following columns:

* **Protected attributes**: 1 to 3 columns holding the attribute values, such as `gender` or `age_band`.
* **Score**: A numeric column, such as `risk_score`.

Column parameters must refer to top-level columns. Fields nested inside a dict
or JSON column are not available, so promote them to their own column first.

## Parameters

| Parameter | Required | Default | Description |
| - | - | - | - |
| `protected_attribute_columns` | Yes | | List of 1 to 3 columns that define the groups. |
| `score_column` | Yes | | Numeric column holding the score. |
| `higher_is_favorable` | No | `true` | Set to `false` when a lower score is the favorable outcome, such as a risk score. |
| `reference_strategy` | No | `highest` | `highest` compares to the group with the highest scoring rate. `named` compares to the group in `reference_value`. |
| `reference_value` | No | | The reference group's value. Required with `named`, which needs exactly one protected attribute column. |
| `impact_ratio_threshold` | No | `0.8` | A group fails when its scoring-rate impact ratio is below this value. Greater than 0 and at most 1. |
| `significance_test` | No | `fisher_exact` | `none`, `z_test`, `fisher_exact`, or `chi_square`. Applied to the share of rows scoring beyond the median. |
| `alpha` | No | `0.05` | Significance level, strictly between 0 and 1. |
| `p_value_adjustment` | No | `none` | `none`, `holm`, or `benjamini_hochberg`. |
| `require_significance` | No | `false` | When `true`, a group fails only if it is also a significant disparity. Needs a `significance_test` other than `none`. |
| `min_group_size` | No | `30` | Minimum rows for a group to be sufficient. |
| `min_group_share` | No | `0.02` | Minimum share of evaluated rows, from 0 to 1, for a group to be sufficient. |

> [!NOTE]
>
> The default `min_group_size` of `30` is stricter than NYC Local Law 144, which
> only lets an audit set aside categories under 2% of the rows. For a literal
> LL144 audit, set `min_group_size` to `1` and `min_group_share` to `0.02` (its
> default). Using this test still does not by itself constitute a bias audit.

## Measurements

Thresholds are set on one of the measurements below.

| Measurement | Description |
| - | - |
| `failingGroupCount` | Number of groups that fail. Use `<= 0` for "no group fails". |
| `failingGroupPercentage` | Share of sufficient groups that fail, as a fraction between 0 and 1. |
| `minScoringRateImpactRatio` | Lowest scoring-rate impact ratio among the non-reference sufficient groups. Typically `>= 0.8`. |
| `maxMeanDifference` | Largest shortfall in mean score vs the reference group, in the score's own units. `0` when no group trails. |
| `maxMedianDifference` | Largest shortfall in median score vs the reference group, in the score's own units. `0` when no group trails. |
| `maxStandardizedMeanDifference` | Largest shortfall in the standardized mean difference vs the reference. A value of `0.2` is conventionally a small effect. |
| `significantDisparityCount` | Number of groups that are significantly disadvantaged compared with the reference. |
| `insufficientGroupCount` | Number of groups below the minimum size or share. |
| `missingAttributeRate` | Share of rows with a usable score excluded because at least one protected attribute was missing. |
| `minGroupShare` | Smallest group's share of the evaluated rows. |

The mean, median, and standardized mean difference measurements count only
shortfalls vs the reference: how far the furthest group sits on its unfavorable
side, which is below the reference when `higher_is_favorable` is `true` and
above it when `false`. A group ahead of the reference is not a disparity and
adds nothing, so each measurement is `0` when no group trails. The per-group
table still shows the signed differences.

Because the mean and median shortfalls depend on the scale of your score,
choose their threshold values for your own score. For a score that runs from 0
to 100, a threshold of `5` allows a shortfall of up to 5 points.

## Test configuration examples

If you are writing a `tests.json`, here are a few valid configurations for the group score disparity test:

```json Development theme={null}
[
  {
    "name": "Risk score parity by gender",
    "description": "Ensures the share of rows scoring below the overall median risk is at least 80% of the best group's share for every gender",
    "type": "fairness",
    "subtype": "groupScoreDisparity",
    "thresholds": [
      {
        "insightName": "groupScoreDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "score_column", "value": "risk_score" },
          { "name": "higher_is_favorable", "value": false }
        ],
        "measurement": "minScoringRateImpactRatio",
        "operator": ">=",
        "value": 0.8
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "b556eb85-1e91-4d6d-9363-21c3168658b4"
  },
  {
    "name": "Risk score effect size by gender",
    "description": "Ensures no gender group's mean risk score is more than 0.2 pooled standard deviations above the reference group's",
    "type": "fairness",
    "subtype": "groupScoreDisparity",
    "thresholds": [
      {
        "insightName": "groupScoreDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "score_column", "value": "risk_score" },
          { "name": "higher_is_favorable", "value": false }
        ],
        "measurement": "maxStandardizedMeanDifference",
        "operator": "<=",
        "value": 0.2
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "40e629c6-be59-4bb3-ad0a-4d180819a4af"
  }
]
```

```json Monitoring theme={null}
[
  {
    "name": "Weekly risk score parity by gender",
    "description": "Fails a gender group only when its scoring rate is below 0.8 of the best group's and Fisher's exact test on the share scoring below the overall median is significant, over a 7-day window",
    "type": "fairness",
    "subtype": "groupScoreDisparity",
    "thresholds": [
      {
        "insightName": "groupScoreDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "score_column", "value": "risk_score" },
          { "name": "higher_is_favorable", "value": false },
          { "name": "significance_test", "value": "fisher_exact" },
          { "name": "require_significance", "value": true }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "275dd8cb-0231-46c9-9f79-fab107d35d7d"
  },
  {
    "name": "Weekly risk score parity by gender and age band",
    "description": "Monitors gender and age band intersections for risk score disparities with Holm correction over a 7-day window",
    "type": "fairness",
    "subtype": "groupScoreDisparity",
    "thresholds": [
      {
        "insightName": "groupScoreDisparity",
        "insightParameters": [
          {
            "name": "protected_attribute_columns",
            "value": ["gender", "age_band"]
          },
          { "name": "score_column", "value": "risk_score" },
          { "name": "higher_is_favorable", "value": false },
          { "name": "p_value_adjustment", "value": "holm" },
          { "name": "require_significance", "value": true },
          { "name": "min_group_size", "value": 20 }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "e0cad967-fd7d-4234-9dac-debbd6c6976e"
  }
]
```

## Related

* [Group outcome disparity test](/tests/catalog/group-outcome-disparity) - Compare favorable outcome rates across protected groups.
* [Group performance parity test](/tests/catalog/group-performance-parity) - Compare error rates and performance metrics across protected groups.
* [Group metric parity test](/tests/catalog/group-metric-parity) - Compare model metrics and column statistics across protected groups.
* [Group by column statistic test](/tests/catalog/group-by-column-statistic) - Check a simple statistic for each value of a column.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.