> ## Documentation Index > Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt > Use this file to discover all available pages before exploring further. # Group performance parity > Learn how to use the group performance parity test to compare error rates and performance metrics across protected groups ## Definition The group performance parity test checks whether a model is equally accurate for every protected group (for example, each gender or age band). For every group it computes the false negative rate, false positive rate, true positive rate, true negative rate, precision, accuracy, and F1 from the model's predictions and the ground truth. You pick one of these as the `parity_metric`, and the test compares each group with a reference group, the group that performs best on that metric once its sample size is allowed for. Each run produces a table with one row per group plus a handful of headline measurements, such as the spread of the false negative rate across groups. You set thresholds on those measurements, and the test passes or fails like any other Openlayer test. You can combine up to 3 protected attributes to test **intersections**. ## Taxonomy * **Task types**: LLM, tabular classification, text classification. * **Availability**: development (Continuously evaluate your models and datasets as you iterate on their versions.) and monitoring (Monitor a model in production, measure its health, check for drifts and set up alerts.). > [!NOTE] > > Tabular regression projects are not supported by this test. Pipelines backed > by BigQuery or Snowflake are not yet supported by the fairness tests. The test > is skipped on them. ## Why it matters * A model can have the same overall approval rate for two groups and still be wrong far more often for one of them. Comparing error rates shows who bears the cost of the mistakes, such as valid requests wrongly denied, which an outcome-rate test cannot see. * Choosing the metric lets you match the harm: the false negative rate when a wrong denial is the costly mistake, the false positive rate when a wrong approval is. * Equalized odds, the larger of the true positive rate spread and the false positive rate spread across groups, summarizes both kinds of error in one number. * The test needs ground truth, so it also tells you when overall accuracy differs by group, even if the error you care about does not. * On a monitoring schedule, the test catches a model that degrades for one group before the overall metrics move. ## How it works The test follows these steps on the rows of each evaluation window: 1. **Drop incomplete rows.** Rows with no prediction or no ground truth are excluded first. Among rows with both, rows with a missing value in any protected attribute column are excluded. A blank or whitespace-only value counts as missing. `missingAttributeRate` is the share of rows with both a prediction and ground truth that were excluded for a missing attribute. 2. **Form groups and set small ones aside.** Rows are grouped by the distinct values of the protected attribute columns. A group is *sufficient* when it has at least `min_group_size` rows and at least `min_group_share` of the evaluated rows. Insufficient groups are listed but never the reference group, never fail, and are left out of the group extremes and failing counts. See [Intersections](/tests/catalog/group-outcome-disparity#intersections) for how this plays out with several attributes. 3. **Count outcomes per group.** A prediction or a ground truth is positive when its value is one of `positive_values`, and negative otherwise, so labels with more than two classes are checked against the positive values you list. This gives each group its true positives, false positives, true negatives, and false negatives. Accuracy is binary, like the other metrics: a row is correct when the prediction and the ground truth are both positive or both not positive, so it is (TP + TN) divided by the rows, not an exact match of the labels. 4. **Compute the metrics.** Each group gets all seven metrics listed below. 5. **Choose the reference group.** Each sufficient group with `parity_metric` defined gets the worse side of a Wilson confidence interval for that metric, at level 1 - `alpha` (95% by default): the upper bound for `fnr` and `fpr`, and the lower bound for the others. The group with the best such bound is the reference, and ties go to the larger group. This way a tiny group with a lucky 0 can't set the bar. 6. **Compare each group to the reference.** The gap is the absolute difference between the group's value and the reference's value, on `parity_metric`. Only a group that trails the reference, with a worse value, can fail or be flagged significant. A group that does better than the reference passes however large its gap. 7. **Test significance.** Each group is tested against the reference group on the counts behind `parity_metric`, as described in [Statistical significance](/tests/catalog/group-outcome-disparity#statistical-significance). The `f1` metric has no single 2x2 table behind it, so it has no significance test. 8. **Decide which groups fail.** A group fails when it trails the reference by more than `max_difference`. When `require_significance` is on, it must also be a statistically significant disparity. Your thresholds are then evaluated against the headline measurements. The test result's **Analysis** tab has three sections. **Group breakdown** shows a summary strip and a **Key insights** card with facts from the run. **Bar graph view** charts each group with its confidence interval. **Table view** lists every group's numbers and has a **Download CSV** button. Selecting a group opens records in the **Data** tab using exact matches on the displayed attribute values and non-empty filters on the measured columns. If the analysis combines equivalent spellings, such as `F` and ` F`, only the displayed spelling appears in the drill-down. Confidence intervals help you interpret uncertainty but do not affect pass or fail. ## Parity metrics | Value | Metric | Better when | | - | - | - | | `fnr` | False negative rate: positives predicted negative, as FN / (TP + FN). Default. | Lower | | `fpr` | False positive rate: negatives predicted positive, as FP / (FP + TN). | Lower | | `tpr` | True positive rate, also called recall: TP / (TP + FN). | Higher | | `tnr` | True negative rate, also called specificity: TN / (FP + TN). | Higher | | `precision` | Precision: TP / (TP + FP). | Higher | | `accuracy` | Binary accuracy, positive vs not positive: (TP + TN) / all rows. | Higher | | `f1` | F1 score: 2TP / (2TP + FP + FN). | Higher | The gap is a difference in rates, so `max_difference` of `0.1` allows a group to be up to 10 percentage points worse than the reference group. ## Calibration If your model produces a probability, set `probability_column` to the column holding the predicted probability of the positive class, with values from 0 to 1. The test then adds, for each group, the mean predicted probability minus the observed positive rate, and reports the largest absolute value across sufficient groups as `maxCalibrationGap`. A group with a large gap has probabilities that systematically over-predict or under-predict the positive class compared with what is observed. The calibration gap is reported and can be thresholded, but it does not affect which groups fail. Calibration needs numeric probabilities between 0 and 1. When the column is not numeric, has values outside that range, or has no values for any group large enough to judge, `maxCalibrationGap` is empty and `calibrationUnavailableReason` says why. The parity verdict still runs. A threshold on `maxCalibrationGap` is then skipped, with that reason as the skip message. ## Required columns To compute this metric, your dataset must contain the following columns: * **Protected attributes**: 1 to 3 columns holding the attribute values, such as `gender` or `age_band`. * **Predictions**: The column holding the model's decision or predicted label. * **Ground truths**: The column holding the correct decision or label. It must be a different column from the predictions. * **Probabilities** (optional): The predicted probability of the positive class, for the calibration gap. Column parameters must refer to top-level columns. Fields nested inside a dict or JSON column are not available, so promote them to their own column first. Values in `positive_values` are matched as text, so `["1"]` matches an integer column holding `1`, and `["true"]` matches a boolean column. An integer written with a decimal point, such as `"1.0"`, matches the same way, and the values `1` and `1.0` in a text attribute column form one group. > [!NOTE] > > The test is skipped, not failed, when it cannot be computed. This happens when > a column is missing or empty in the window, when neither the predictions nor > the ground truths contain a positive value, when fewer than 2 groups are > sufficient, when fewer than 2 sufficient groups have the `parity_metric` > defined (for example, `fnr` needs rows whose ground truth is positive), and > when the protected attributes form more than 1,000 groups. The skip message > says which. ## Parameters | Parameter | Required | Default | Description | | - | - | - | - | | `protected_attribute_columns` | Yes | | List of 1 to 3 columns that define the groups. | | `prediction_column` | Yes | | Column holding the model's prediction. | | `ground_truth_column` | Yes | | Column holding the ground truth. Must differ from `prediction_column`. | | `positive_values` | Yes | | List of values that count as the positive class. | | `parity_metric` | No | `fnr` | `fnr`, `fpr`, `tpr`, `tnr`, `precision`, `accuracy`, or `f1`. | | `max_difference` | No | `0.1` | A group fails when it trails the reference by more than this value, from 0 to 1. | | `significance_test` | No | `fisher_exact` | `none`, `z_test`, `fisher_exact`, or `chi_square`. | | `alpha` | No | `0.05` | Significance level, strictly between 0 and 1. | | `p_value_adjustment` | No | `none` | `none`, `holm`, or `benjamini_hochberg`. | | `require_significance` | No | `false` | When `true`, a group fails only if it is also a significant disparity. Needs a `significance_test` other than `none`, and is not available with `f1`. | | `min_group_size` | No | `30` | Minimum rows for a group to be sufficient. | | `min_group_share` | No | `0.02` | Minimum share of evaluated rows, from 0 to 1, for a group to be sufficient. | | `probability_column` | No | | Column holding the predicted probability of the positive class, for `maxCalibrationGap`. | ## Measurements Thresholds are set on one of the measurements below. Every `max...Difference` measurement is computed on every run, whatever the `parity_metric` is. Each one is the difference between the highest and the lowest value of that metric across sufficient groups, whichever group is the reference. So a spread can exceed `max_difference` while no group fails, for example when a group does better than the reference. The `parity_metric` only decides which groups are counted as failing. | Measurement | Description | | - | - | | `failingGroupCount` | Number of groups that fail. Use `<= 0` for "no group fails". | | `failingGroupPercentage` | Share of sufficient groups that fail, as a fraction between 0 and 1. | | `maxFnrDifference` | Spread of the false negative rate across sufficient groups. Typically `<= 0.1`. | | `maxFprDifference` | Spread of the false positive rate across sufficient groups. Typically `<= 0.1`. | | `maxTprDifference` | Spread of the true positive rate across sufficient groups. Typically `<= 0.1`. | | `maxTnrDifference` | Spread of the true negative rate across sufficient groups. Typically `<= 0.1`. | | `maxPrecisionDifference` | Spread of precision across sufficient groups. Typically `<= 0.1`. | | `maxAccuracyDifference` | Spread of accuracy across sufficient groups. Typically `<= 0.1`. | | `maxF1Difference` | Spread of F1 across sufficient groups. Typically `<= 0.1`. | | `equalizedOddsDifference` | The larger of `maxTprDifference` and `maxFprDifference`. Typically `<= 0.1`. | | `maxCalibrationGap` | Largest absolute calibration gap across sufficient groups. Only available with `probability_column`. | | `significantDisparityCount` | Number of groups that trail the reference by a statistically significant margin. | | `insufficientGroupCount` | Number of groups below the minimum size or share. | | `missingAttributeRate` | Share of rows with a prediction and ground truth excluded because a protected attribute was missing. | | `minGroupShare` | Smallest group's share of the evaluated rows. | ## Test configuration examples If you are writing a `tests.json`, here are a few valid configurations for the group performance parity test: ```json Development theme={null} [ { "name": "False negative rate parity by gender", "description": "Ensures no gender group's false negative rate is more than 10 percentage points above the reference group's", "type": "fairness", "subtype": "groupPerformanceParity", "thresholds": [ { "insightName": "groupPerformanceParity", "insightParameters": [ { "name": "protected_attribute_columns", "value": ["gender"] }, { "name": "prediction_column", "value": "decision" }, { "name": "ground_truth_column", "value": "correct_decision" }, { "name": "positive_values", "value": ["APPROVE"] }, { "name": "parity_metric", "value": "fnr" } ], "measurement": "failingGroupCount", "operator": "<=", "value": 0 } ], "subpopulationFilters": null, "mode": "development", "usesValidationDataset": true, "usesTrainingDataset": false, "usesMlModel": false, "syncId": "0e13feb8-9aa5-4098-a0c5-a102cc818ccd" }, { "name": "Equalized odds by gender", "description": "Ensures the true positive rate and false positive rate gaps between genders stay at or below 0.1", "type": "fairness", "subtype": "groupPerformanceParity", "thresholds": [ { "insightName": "groupPerformanceParity", "insightParameters": [ { "name": "protected_attribute_columns", "value": ["gender"] }, { "name": "prediction_column", "value": "decision" }, { "name": "ground_truth_column", "value": "correct_decision" }, { "name": "positive_values", "value": ["APPROVE"] } ], "measurement": "equalizedOddsDifference", "operator": "<=", "value": 0.1 } ], "subpopulationFilters": null, "mode": "development", "usesValidationDataset": true, "usesTrainingDataset": false, "usesMlModel": false, "syncId": "2f2aba81-7879-4a82-99af-8e3c099cda4c" } ] ``` ```json Monitoring theme={null} [ { "name": "Weekly false negative rate parity by gender", "description": "Fails a gender group only when its false negative rate is more than 10 percentage points above the reference group's and the gap is significant, over a 7-day window", "type": "fairness", "subtype": "groupPerformanceParity", "thresholds": [ { "insightName": "groupPerformanceParity", "insightParameters": [ { "name": "protected_attribute_columns", "value": ["gender"] }, { "name": "prediction_column", "value": "decision" }, { "name": "ground_truth_column", "value": "correct_decision" }, { "name": "positive_values", "value": ["APPROVE"] }, { "name": "parity_metric", "value": "fnr" }, { "name": "max_difference", "value": 0.1 }, { "name": "require_significance", "value": true } ], "measurement": "failingGroupCount", "operator": "<=", "value": 0 } ], "subpopulationFilters": null, "mode": "monitoring", "usesProductionData": true, "evaluationWindow": 604800, "delayWindow": 0, "syncId": "c033ede2-35bc-4b78-aaa4-5bcb1a22b03c" }, { "name": "Weekly accuracy parity by gender and age band", "description": "Fails a gender and age band intersection only when its accuracy is more than 10 percentage points below the reference group's and the gap is significant after Holm correction, over a 7-day window", "type": "fairness", "subtype": "groupPerformanceParity", "thresholds": [ { "insightName": "groupPerformanceParity", "insightParameters": [ { "name": "protected_attribute_columns", "value": ["gender", "age_band"] }, { "name": "prediction_column", "value": "decision" }, { "name": "ground_truth_column", "value": "correct_decision" }, { "name": "positive_values", "value": ["APPROVE"] }, { "name": "parity_metric", "value": "accuracy" }, { "name": "p_value_adjustment", "value": "holm" }, { "name": "require_significance", "value": true }, { "name": "min_group_size", "value": 20 } ], "measurement": "failingGroupCount", "operator": "<=", "value": 0 } ], "subpopulationFilters": null, "mode": "monitoring", "usesProductionData": true, "evaluationWindow": 604800, "delayWindow": 0, "syncId": "f2535244-d27c-4c1b-993d-88c8e877face" } ] ``` ## Related * [Group outcome disparity test](/tests/catalog/group-outcome-disparity) - Compare favorable outcome rates across protected groups. * [Group score disparity test](/tests/catalog/group-score-disparity) - Compare a numeric score across protected groups. * [Group metric parity test](/tests/catalog/group-metric-parity) - Compare model metrics and column statistics across protected groups. * [False positive rate test](/tests/catalog/false-positive-rate) - Measure the overall false positive rate. * [Recall test](/tests/catalog/recall) - Measure ability to find all positive instances. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.