You maybe know the LIME algorithm from some of my earlier blog posts. It can be quite useful to “debug” data sets and understand machine learning models better. But LIME is fooled very easily. We use the eli5
TextExplainer which is based on LIME and the 20newsgroup data set to show how LIME can fail.
import numpy as np from sklearn.datasets import fetch_20newsgroups from eli5.lime import TextExplainer # we only look at the case atheism vs. copmuter graphics data = fetch_20newsgroups(categories=['alt.atheism', 'comp.graphics'], remove=['footers']) X, y = data.data, data.target te = TextExplainer(random_state=42, n_samples=10000)
Now we code a fairly stupid blackbox model, that just uses the length of the longest token in the document as a feature and splits the data based on this feature.
class BlackBoxModel(): def fit(self, X=None, y=None): pass def predict(self, X, y=None): return np.array([int(max([len(x_ii) for x_ii in x_i.split(" ")]) >= 27) for x_i in X]) def predict_proba(self, X, y=None): return np.array([ 1.0-self.predict(X), self.predict(X)-0.0 ]).T
model = BlackBoxModel() model.fit(X, y)
Let’s explain the predictions of our black-box model:
From: af774@cleveland.Freenet.Edu (Chad Cipiti) Subject: Good shareware paint and/or animation software for SGI? Organization: Case Western Reserve University, Cleveland, OH (USA) Lines: 15 Reply-To: af774@cleveland.Freenet.Edu (Chad Cipiti) NNTP-Posting-Host: hela.ins.cwru.edu Does anyone know of any good shareware animation or paint software for an SGI machine? I've exhausted everyplace on the net I can find and still don't hava a nice piece of software. Thanks alot! Chad
This is clearly talking about computer graphics here. So let’s see what our blackbox model predicts and how LIME would explain it.
y_pred = model.predict([X]) print(data.target_names[y_pred])
So the prediction is corrent, but let’s see if we can understand why.
te.fit(X, model.predict_proba) te.show_prediction(target_names=data.target_names)
y=comp.graphics (probability 0.999, score 6.993) top features
|+7.976||Highlighted in text (sum)|
from: firstname.lastname@example.org (chad cipiti) subject: good shareware paint and/or animation software for sgi? organization: case western reserve university, cleveland, oh (usa) lines: 15 reply–to: email@example.com (chad cipiti) nntp–posting–host: hela.ins.cwru.edu does anyone know of any good shareware animation or paint software for an sgi machine? i‘ve exhausted everyplace on the net i can find and still don‘t hava a nice piece of software. thanks alot! chad
So we would conclude, that email addresses and words like software and university are important, but we know they are not. This issue always arises, when you are not sure what kind of features your model internally uses. So, for example, it can also be present in more subtle ways in large language models like BERT. You can find out more on this issue on the eli5 website. Note, that it is fairly easy to rationalize this explanation as a human if you are note really careful. This is what makes Explainable AI so dangerous. So be aware of these issues and apply these methods carefully.
You might also be interested in:
- Debugging black-box text classifiers with LIME: Explainable AI series: Debugging black-box text classifiers with LIME How the LIME algorithm fails …
- Explain neural networks with keras and eli5: Explainable AI series: Debugging black-box text classifiers with LIME How the LIME algorithm fails …
- Interpretable Named entity recognition with keras and LIME: Named entity recognition series: Introduction To Named Entity Recognition In Python Named Entity Re …