Exploring explainability

Local and global explanations

So far in the tutorial, we’ve been conducting error analysis mainly by identifying slices of the data over which our model does not perform as well as we desired. However, so far, we haven’t asked a fundamental question: why does our model behave the way it does?

We will use explainability techniques to understand some of the driving forces behind our model’s predictions.

In broad strokes, explainability techniques help justify our model’s predictions. These explanations can be local or global, providing a distinct perspective to practitioners and businesses. Let’s explore these two layers of explainability for our churn classification model from the bottom up.

Local explanations

Local explanations provide insights into individual model predictions.

Using our urgent event classifier, local explanations help us answer the question of why did our model classify a specific message as urgent or not urgent?

To look at local explanations, click on any row of the data shown below the Error analysis panel. With Openlayer, you can access local explanations for all of the model’s predictions, powered by LIME, one of the most popular model-agnostic explainability techniques.

Let’s pick a row from the problematic error slice we have identified in the previous section, namely, the slice of data where our model predicts Not urgent, but for which the ground truth is Urgent.

Each token receives a score. Values shown in shades of green (which are not present in this data sample) indicate the tokens that contributed to the model’s prediction toward the correct direction. Values shown in shades of red indicate the tokens that contributed negatively to the model’s prediction, pushing it in the wrong direction. Therefore, it is important to remember that these values are always relative to the true label.

In this specific example, the true label is Urgent, but our model predicted it was a Not urgent.

At the end of the day, the model’s prediction is a balance between features that push it in the right direction and features that nudge it in the wrong direction.

Error analysis must be a scientific process, incorporating hypothesizing and experimenting to its core. That’s one of the roles of the what-if analysis.

To conduct a what-if analysis with local explanations, we can simply modify the sentence by clicking on Edit row and then clicking on What-if at the bottom of the page. For example, what would our model’s prediction be if we changed the location from Cul-de-Sac to Port-au-Prince?

Now we can directly compare the two explanations. Notice that if we simply change the location, the model gets it right.

What if?

Feel free to explore some local explanations. Can you use the what-if analysis and slightly modify sentences to flip our model’s prediction in other rows?


Sensitivity to the message's location

Notice in the previous example that we could completely flip the model’s prediction just by modifying the location. This is possibly a symptom of certain locations not being as well represented as others in the training set.


Actionable insights

Using local explanations, practitioners can get to the root cause of problematic predictions their models are making. Furthermore, they can build confidence that the model is considering reasonable data to make its predictions and not simply over-indexing to certain tokens.

After exploring other rows, it is possible to note that such behavior is widespread and drives the bulk of the samples that fit into the largest error class. A lot of the Urgent messages from the training set come from the disaster response to the earthquake that happened in Haiti, in 2010. Furthermore, a lot of them are from the capital of Haiti, Port-au-Prince. Therefore, it looks like our model is over-indexing to that location.

It is clear, then, that to boost our model’s performance and shrink the most critical error class, we need to increase our model’s robustness to locations. We will solve this problem in one of the last parts of the tutorial.

Global explanations

Now we move one layer up to global explanations.

Global explanations are built by aggregating local explanations and help us understand which tokens and stopwords contributed the most to the (mis)predictions made by the model over a data cohort or the whole dataset.

For example, for the messages about the weather, what were the tokens and stopwords that contributed the most to our model’s mispredictions? What about our model’s correct predictions?

These kinds of questions can be easily answered with Openlayer.

Let’s now dive into the other error class our model is making, namely, the slice of data where our model classifies a message as Urgent, but for which the ground truth is Not urgent. What token(s) are possibly behind such behavior?

The answer will be shown in the Feature importance section of the error analysis panel. However, we need to first filter the data cohort that we are interested in explaining.

Identifying the most mispredictive tokens

First, filter the data to show only the rows our model predicted as Urgent, but for which the label is Not urgent. Then, head to the Feature importance section on the error analysis panel to look at the most predictive and mispredictive tokens.

When you filter data cohorts, what we see in the Feature importance section are the most predictive and most mispredictive tokens and stopwords for that specific data cohort.

Filtering by mispredictive tokens

Click on one of the blocks shown to see what happens. Did you notice what happened to the data shown below the Error analysis panel?

When we click on one of the blocks shown, the data shown below the Error analysis panel displays only the rows that fall within that category.


Most mispredictive expressions

The previous image tells us that in most of the rows where our model made the mistake of predicting Urgent when the true label was Not urgent, the tokens “need” and “help” were some of the main culprits.

When the model classifies a message that has the tokens “need” and “help”, it is inclined to put it into the Urgent bucket. Indeed, a lot of the messages with these tokens are urgent, from people needing help. However, this is not always the case. For example, the messages "I need help with finding the area of shaded polygons” and “I need to do an easy fast science fair project or I’ll flunk my class can y’all please help me” are clearly not urgent, but they are fooling our model.


Actionable insights

The model is, in a sense, paying too much attention to certain expressions that tend to be associated with Urgent messages. To shrink this error class, we need to increase our model’s robustness to these and other expressions, possibly making use of synthetic data with paraphrases of the problematic expressions, such as “I need to”, “I want to”, “I’d like to”, and others.

What if?

Pick one of the rows where the model makes the mistake mentioned above. Can you flip its prediction by paraphrasing the sentence?

In the next part of the tutorial, we will zoom into our model’s predictions as we conduct an error cohort analysis and document error patterns.