Group score disparity

Definition

The group score disparity test compares a numeric score, such as a risk score or a ranking score, across protected groups (for example, each gender or age band). For every group it computes the mean, median, and scoring rate, which is the share of the group’s rows scoring above the overall median. It then compares each group with a reference group using the scoring-rate impact ratio, the mean and median gaps, and a standardized mean difference, and it can test whether the gap is statistically significant. Each run produces a table with one row per group plus a handful of headline measurements. You set thresholds on those measurements, and the test passes or fails like any other Openlayer test. You can combine up to 3 protected attributes to test intersections.

Taxonomy

Task types: LLM, tabular classification, tabular regression, text classification.
Availability: and .

Pipelines backed by BigQuery or Snowflake are not yet supported by the fairness tests. The test is skipped on them.

Why it matters

Scores often drive decisions indirectly, by ranking candidates, setting a price, or deciding who gets a review. A gap in scores between groups can produce unequal outcomes even when no single decision looks unfair.
The scoring rate, the share of a group scoring above the overall median, follows the definition used in bias audits of automated employment decision tools, such as those under NYC Local Law 144. It turns a continuous score into a ratio that can be screened with the same four-fifths rule as a selection rate. Using this test does not by itself constitute a bias audit.
The mean, median, and standardized mean difference show how large the gap is in the score’s own units, which a ratio alone can hide.
On a monitoring schedule, the test catches score drift that affects some groups more than others.

The four-fifths flag is an adverse-impact indicator: a screening heuristic that tells you where to look. It is not a legal determination. A ratio above or below 0.8 is not, on its own, evidence that a model is or is not discriminatory.

How it works

The test follows these steps on the rows of each evaluation window:

Drop incomplete rows. Rows with no usable finite score are excluded first. Among rows with a usable score, rows with a missing value in any protected attribute column are excluded. A blank or whitespace-only value counts as missing. missingAttributeRate is the share of rows with a usable score that were excluded for a missing attribute.
Form groups and set small ones aside. Rows are grouped by the distinct values of the protected attribute columns. A group is sufficient when it has at least min_group_size rows and at least min_group_share of the evaluated rows. Insufficient groups are listed but never the reference group, never fail, and are left out of the group extremes and failing counts. See Intersections for how this plays out with several attributes.
Compute the scoring rate. The overall median is taken across all evaluated rows. The scoring rate of a group is the share of its rows with a score above that median, or below it when higher_is_favorable is false. Rows exactly at the median do not count.
Choose the reference group. With highest (the default) it is the sufficient group with the highest scoring rate. With named it is the group you set in reference_value.
Compare each group to the reference. The scoring-rate impact ratio is the group’s scoring rate divided by the reference’s. The mean and median differences are relative to the reference group, and the standardized mean difference is the mean difference divided by the pooled standard deviation of the two groups. Every sufficient group with a scoring-rate impact ratio below 0.8 gets the four-fifths flag, whatever impact_ratio_threshold is.
Test significance. Each group is compared with the reference group on the share of rows scoring above the overall median, the same share the impact ratio is built on. This is a median test: it judges that share, not the whole score distribution. The significance_test parameter picks the test, and fisher_exact is the default. Use p_value_adjustment to correct for comparing many groups. See Statistical significance for the tests and corrections.
Decide which groups fail. A group fails when its scoring-rate impact ratio is below impact_ratio_threshold. When require_significance is on, it must also be a statistically significant disparity, meaning its adjusted p-value is below alpha and its scoring rate is below the reference’s.

Your thresholds are then evaluated against the headline measurements. The test result’s Analysis tab has three sections. Group breakdown shows a summary strip and a Key insights card with facts from the run. Bar graph view charts each group with its confidence interval. Table view lists every group’s numbers and has a Download CSV button. Selecting a group opens records in the Data tab using exact matches on the displayed attribute values and non-empty filters on the measured columns. If the analysis combines equivalent spellings, such as F and F, only the displayed spelling appears in the drill-down. Confidence intervals help you interpret uncertainty but do not affect pass or fail.

The test is skipped, not failed, when it cannot be computed. This happens when a column is missing or empty in the window, when the score column is not numeric or has only infinite values, when fewer than 2 groups are sufficient, when the reference group has no scores beyond the median, when a named reference group has no rows or falls below the minimum size or share, and when the protected attributes form more than 1,000 groups. A reference group has no scores beyond the median when scores are heavily tied, such as a 0/1 score where most rows are 1, or most rows are 0 when higher_is_favorable is false. For that kind of score, use the Group outcome disparity test instead. The skip message says which.

Required columns

To compute this metric, your dataset must contain the following columns:

Protected attributes: 1 to 3 columns holding the attribute values, such as gender or age_band.
Score: A numeric column, such as risk_score.

Column parameters must refer to top-level columns. Fields nested inside a dict or JSON column are not available, so promote them to their own column first.

Parameters

Parameter	Required	Default	Description
`protected_attribute_columns`	Yes		List of 1 to 3 columns that define the groups.
`score_column`	Yes		Numeric column holding the score.
`higher_is_favorable`	No	`true`	Set to `false` when a lower score is the favorable outcome, such as a risk score.
`reference_strategy`	No	`highest`	`highest` compares to the group with the highest scoring rate. `named` compares to the group in `reference_value`.
`reference_value`	No		The reference group’s value. Required with `named`, which needs exactly one protected attribute column.
`impact_ratio_threshold`	No	`0.8`	A group fails when its scoring-rate impact ratio is below this value. Greater than 0 and at most 1.
`significance_test`	No	`fisher_exact`	`none`, `z_test`, `fisher_exact`, or `chi_square`. Applied to the share of rows scoring beyond the median.
`alpha`	No	`0.05`	Significance level, strictly between 0 and 1.
`p_value_adjustment`	No	`none`	`none`, `holm`, or `benjamini_hochberg`.
`require_significance`	No	`false`	When `true`, a group fails only if it is also a significant disparity. Needs a `significance_test` other than `none`.
`min_group_size`	No	`30`	Minimum rows for a group to be sufficient.
`min_group_share`	No	`0.02`	Minimum share of evaluated rows, from 0 to 1, for a group to be sufficient.

The default min_group_size of 30 is stricter than NYC Local Law 144, which only lets an audit set aside categories under 2% of the rows. For a literal LL144 audit, set min_group_size to 1 and min_group_share to 0.02 (its default). Using this test still does not by itself constitute a bias audit.

Measurements

Thresholds are set on one of the measurements below.

Measurement	Description
`failingGroupCount`	Number of groups that fail. Use `<= 0` for “no group fails”.
`failingGroupPercentage`	Share of sufficient groups that fail, as a fraction between 0 and 1.
`minScoringRateImpactRatio`	Lowest scoring-rate impact ratio among the non-reference sufficient groups. Typically `>= 0.8`.
`maxMeanDifference`	Largest shortfall in mean score vs the reference group, in the score’s own units. `0` when no group trails.
`maxMedianDifference`	Largest shortfall in median score vs the reference group, in the score’s own units. `0` when no group trails.
`maxStandardizedMeanDifference`	Largest shortfall in the standardized mean difference vs the reference. A value of `0.2` is conventionally a small effect.
`significantDisparityCount`	Number of groups that are significantly disadvantaged compared with the reference.
`insufficientGroupCount`	Number of groups below the minimum size or share.
`missingAttributeRate`	Share of rows with a usable score excluded because at least one protected attribute was missing.
`minGroupShare`	Smallest group’s share of the evaluated rows.

The mean, median, and standardized mean difference measurements count only shortfalls vs the reference: how far the furthest group sits on its unfavorable side, which is below the reference when higher_is_favorable is true and above it when false. A group ahead of the reference is not a disparity and adds nothing, so each measurement is 0 when no group trails. The per-group table still shows the signed differences. Because the mean and median shortfalls depend on the scale of your score, choose their threshold values for your own score. For a score that runs from 0 to 100, a threshold of 5 allows a shortfall of up to 5 points.

Test configuration examples

If you are writing a tests.json, here are a few valid configurations for the group score disparity test:

[
  {
    "name": "Risk score parity by gender",
    "description": "Ensures the share of rows scoring below the overall median risk is at least 80% of the best group's share for every gender",
    "type": "fairness",
    "subtype": "groupScoreDisparity",
    "thresholds": [
      {
        "insightName": "groupScoreDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "score_column", "value": "risk_score" },
          { "name": "higher_is_favorable", "value": false }
        ],
        "measurement": "minScoringRateImpactRatio",
        "operator": ">=",
        "value": 0.8
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "b556eb85-1e91-4d6d-9363-21c3168658b4"
  },
  {
    "name": "Risk score effect size by gender",
    "description": "Ensures no gender group's mean risk score is more than 0.2 pooled standard deviations above the reference group's",
    "type": "fairness",
    "subtype": "groupScoreDisparity",
    "thresholds": [
      {
        "insightName": "groupScoreDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "score_column", "value": "risk_score" },
          { "name": "higher_is_favorable", "value": false }
        ],
        "measurement": "maxStandardizedMeanDifference",
        "operator": "<=",
        "value": 0.2
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "40e629c6-be59-4bb3-ad0a-4d180819a4af"
  }
]

[
  {
    "name": "Weekly risk score parity by gender",
    "description": "Fails a gender group only when its scoring rate is below 0.8 of the best group's and Fisher's exact test on the share scoring below the overall median is significant, over a 7-day window",
    "type": "fairness",
    "subtype": "groupScoreDisparity",
    "thresholds": [
      {
        "insightName": "groupScoreDisparity",
        "insightParameters": [
          { "name": "protected_attribute_columns", "value": ["gender"] },
          { "name": "score_column", "value": "risk_score" },
          { "name": "higher_is_favorable", "value": false },
          { "name": "significance_test", "value": "fisher_exact" },
          { "name": "require_significance", "value": true }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "275dd8cb-0231-46c9-9f79-fab107d35d7d"
  },
  {
    "name": "Weekly risk score parity by gender and age band",
    "description": "Monitors gender and age band intersections for risk score disparities with Holm correction over a 7-day window",
    "type": "fairness",
    "subtype": "groupScoreDisparity",
    "thresholds": [
      {
        "insightName": "groupScoreDisparity",
        "insightParameters": [
          {
            "name": "protected_attribute_columns",
            "value": ["gender", "age_band"]
          },
          { "name": "score_column", "value": "risk_score" },
          { "name": "higher_is_favorable", "value": false },
          { "name": "p_value_adjustment", "value": "holm" },
          { "name": "require_significance", "value": true },
          { "name": "min_group_size", "value": 20 }
        ],
        "measurement": "failingGroupCount",
        "operator": "<=",
        "value": 0
      }
    ],
    "subpopulationFilters": null,
    "mode": "monitoring",
    "usesProductionData": true,
    "evaluationWindow": 604800,
    "delayWindow": 0,
    "syncId": "e0cad967-fd7d-4234-9dac-debbd6c6976e"
  }
]

Group outcome disparity test - Compare favorable outcome rates across protected groups.
Group performance parity test - Compare error rates and performance metrics across protected groups.
Group metric parity test - Compare model metrics and column statistics across protected groups.
Group by column statistic test - Check a simple statistic for each value of a column.

Get started

Workspace setup

Governance

Observability

Offline testing

Tests

Gateway

Data quality monitoring

Administration

Notifications

Other resources

Group score disparity

Definition

Taxonomy

Why it matters

How it works

Required columns

Parameters

Measurements

Test configuration examples

​Definition

​Taxonomy

​Why it matters

​How it works

​Required columns

​Parameters

​Measurements

​Test configuration examples

​Related

Definition

Taxonomy

Why it matters

How it works

Required columns

Parameters

Measurements

Test configuration examples

Related