Exploring explainability

Local and global explanations

So far in the tutorial, we’ve been conducting error analysis mainly by identifying slices of the data over which our model does not perform as well as we desired. However, so far, we haven’t asked a fundamental question: why does our model behave the way it does?

We will use explainability techniques to understand some of the driving forces behind our model’s predictions.

In broad strokes, explainability techniques help justify our model’s predictions. These explanations can be local or global, providing a distinct perspective to practitioners and businesses. Let’s explore these two layers of explainability for our churn classification model from the bottom up.

Local explanations

Local explanations provide insights into individual model predictions.

Using our churn classifier, local explanations help us answer the question of why did our model predict that a specific user would churn?

To look at local explanations, click on any row of the data shown below the Error analysis panel. With Openlayer, you can access local explanations for all of the model’s predictions, powered by LIME and SHAP, two of the most popular model-agnostic explainability techniques.

Let’s now understand what we see.

Each feature receives a score. Values shown in shades of green (which are not present in this data sample) indicate the features that contributed to the model’s prediction toward the correct direction. Values shown in shades of red indicate the features that contributed negatively to the model’s prediction, pushing it in the wrong direction. Therefore, it is important to remember that these values are always relative to the true label.

In this specific example, the true label is Retained, but our model predicted it was an Exited user. What we can see from the explainability scores is that Age was the feature that strongly contributed to our model’s mistake in this case.

At the end of the day, the model’s prediction is a balance between features that push it in the right direction and features that nudge it in the wrong direction.

In the previous image, we see the feature scores calculated using LIME. If you’d like to see the scores computed by SHAP, you can just click on Show SHAP values to toggle between the two.

Error analysis must be a scientific process, incorporating hypothesizing and experimenting to its core. That’s one of the roles of the what-if analysis.

To conduct a what-if analysis with local explanations, we can simply modify some of the feature values right below the Comparison run and click on What-if at the bottom of the page. For example, what would our model’s prediction be if the user was just a bit younger, all other features being equal?

Now we can directly compare the two explanations. Notice that if the user was just a bit younger, 43 instead of 46, the model would get it right. That’s why the age is now shown in green.

What if?

Feel free to explore some local explanations. Can you use the what-if analysis and play with the feature values to flip our model’s prediction in other rows?


Sensitivity to the feature Age

Notice in the previous example that we could completely flip the model’s prediction just by modifying the user’s age in 3 years. This is possibly a symptom of certain age groups not being as well represented as others in the training set.


Actionable insights

Using local explanations, practitioners can get to the root cause of problematic predictions their models are making. Furthermore, they can build confidence that the model is considering reasonable data to make its predictions and not simply over-indexing to certain features.

Global explanations

Now we move one layer up to global explanations.

Global explanations are built by aggregating local explanations and help us understand which features contributed the most to the (mis)predictions made by the model over a data cohort or the whole dataset.

For example, for the users aged between 25 and 35, what were the features that contributed the most to our model’s mispredictions? What about our model’s correct predictions?

These kinds of questions can be easily answered with Openlayer.

Let’s get back to the problematic slice we have identified in the previous section, namely, the slice of data where our model predicts users as Exited, but for which the ground truth is Retained. What feature(s) are possibly behind such behavior?

The answer will be shown in the Feature importance tab of the error analysis panel. However, we need to first filter the data cohort that we are interested in explaining.

Identifying the most mispredictive features

First, filter the data to show only the rows our model predicted as Exited, but for which the label is Retained. Then, head to the Feature importance section on the error analysis panel to look at the most predictive and mispredictive features.

When you filter data cohorts, what we see in the Feature importance tab are the most predictive and most mispredictive features for that specific data cohort.


Mispredictive feature ranges

Click on one of the blocks shown to see what happens. Did you notice what happened to the data shown below the Error analysis panel?

When we click on one of the blocks shown, two things happen. First, the data shown below the Error analysis panel displays only the rows that fall within that category. Second, the Error analysis panel is split in two so that we can dive even deeper into understanding our model’s predictions. When we click on Gender, for example, which was pointed out as the most mispredictive feature for that data cohort, we can see on the right-hand part of the error analysis panel the clusters of feature values.


Gender feature values

The previous image tells us that in most of the rows where our model made the mistake of predicting Exited when the true label was Retained, the feature Gender was one of the main culprits.
Furthermore, in all of these cases, Gender was equal to “Female.” This means that most of the time our model tries to predict a sample from a female user, it doesn’t know what to do!


Actionable insights

A problematic behavior such as the one we have identified is possibly a symptom of a training set with few samples from female users. More specifically, based on the type of mistake our model makes, it seems like there are few samples from Retained females in the training set.
In that case, since our model hasn’t been exposed to many such examples during training, it ends up making a lot of mistakes.

A quick check of the training set reveals that that’s indeed the case: we only have 100 samples where “Gender” is equal to “Female” out of a total of 4100 samples on the training set. Furthermore, out of the female users on the training set, only 20% were Retained. This raises serious questions about the origin of such bias in the dataset and the corrective actions needed to address it.

In the next part of the tutorial, we will zoom into our model’s predictions for each gender and look at another consequence of the unbalanced dataset as we conduct an error cohort analysis.