You maybe know the LIME algorithm from some of my earlier blog posts. It can be quite useful to “debug” data sets and understand machine learning models better. But LIME is fooled very easily. We use the eli5
TextExplainer which is based on LIME and the 20newsgroup dataset to make my point.
import numpy as np from sklearn.datasets import fetch_20newsgroups from eli5.lime import TextExplainer # we only look at the case atheism vs. copmuter graphics data = fetch_20newsgroups(categories=['alt.atheism', 'comp.graphics'], remove=['footers']) X, y = data.data, data.target te = TextExplainer(random_state=42, n_samples=10000)
Now we code a fairly stupid blackbox model, that just uses the length of the longest token in the document as a feature and splits the data based on this feature.
class BlackBoxModel(): def fit(self, X=None, y=None): pass def predict(self, X, y=None): return np.array([int(max([len(x_ii) for x_ii in x_i.split(" ")]) >= 27) for x_i in X]) def predict_proba(self, X, y=None): return np.array([ 1.0-self.predict(X), self.predict(X)-0.0 ]).T
model = BlackBoxModel() model.fit(X, y)
Let’s explain the predictions of our black-box model:
From: af774@cleveland.Freenet.Edu (Chad Cipiti) Subject: Good shareware paint and/or animation software for SGI? Organization: Case Western Reserve University, Cleveland, OH (USA) Lines: 15 Reply-To: af774@cleveland.Freenet.Edu (Chad Cipiti) NNTP-Posting-Host: hela.ins.cwru.edu Does anyone know of any good shareware animation or paint software for an SGI machine? I've exhausted everyplace on the net I can find and still don't hava a nice piece of software. Thanks alot! Chad
This is clearly talking about computer graphics here. So let’s see what our blackbox model predicts and how LIME would explain it.
y_pred = model.predict([X]) print(data.target_names[y_pred])
So the prediction is corrent, but let’s see if we can understand why.
te.fit(X, model.predict_proba) te.show_prediction(target_names=data.target_names)
y=comp.graphics (probability 0.999, score 6.993) top features
|+7.976||Highlighted in text (sum)|
from: email@example.com (chad cipiti) subject: good shareware paint and/or animation software for sgi? organization: case western reserve university, cleveland, oh (usa) lines: 15 reply-to: firstname.lastname@example.org (chad cipiti) nntp-posting-host: hela.ins.cwru.edu does anyone know of any good shareware animation or paint software for an sgi machine? i've exhausted everyplace on the net i can find and still don't hava a nice piece of software. thanks alot! chad
So we would conclude, that email addresses and words like software and university are important, but we know they are not. More on this issue can be found on the eli5 website. Note, that it is fairly easy to rationalize this explanation as a human if you are note really careful. This is what makes Explainable AI so dangerous.