A walkthrough of text analysis and TF-IDF#
We'll start by using scikit-learn to count words, then come across some of the issues with simple word count analysis. Most of these problems can be tackled with TF-IDF - a single word might mean less in a longer text, and common words may contribute less to meaning than more rare ones.
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
import re
from nltk.stem.porter import PorterStemmer
pd.options.display.max_columns = 30
%matplotlib inline
Text analysis refresher#
Text analysis has a few parts. We are going to use bag of words analysis, which just treats a sentence like a bag of words - no particular order or anything. It's simple but it usually gets the job done adequately.
Here is our text.
texts = [
"Penny bought bright blue fishes.",
"Penny bought bright blue and orange fish.",
"The cat ate a fish at the store.",
"Penny went to the store. Penny ate a bug. Penny saw a fish.",
"It meowed once at the bug, it is still meowing at the bug and the fish",
"The cat is at the fish store. The cat is orange. The cat is meowing at the fish.",
"Penny is a fish"
]
When you process text, you have a nice long series of steps, but let's say you're interested in three things:
- Tokenizing converts all of the sentences/phrases/etc into a series of words, and then it might also include converting it into a series of numbers - math stuff only works with numbers, not words. So maybe 'cat' is 2 and 'rug' is 4 and stuff like that.
- Counting takes those words and sees how many there are (obviously) - how many times does
meowappear? - Normalizing takes the count and makes new numbers - maybe it's how many times
meowappears vs. how many total words there are, or maybe you're seeing how oftenmeowcomes up to see whether it's important.
"Penny bought bright blue fishes".split()
Penny bought bright blue fishes.
If we tokenize that sentence, we're just lowercasing it, removing the punctuation and splitting on spaces - penny bought bright blue fishes. It also works for other languages:
у меня зазвонил телефон
That's Russian for "my phone is ringing." It works just as well with the tokenizer we used for English - lowercase it, remove punctuation, split on spaces. No big deal!
私はえんぴつです。
This is Japanese for "I am a pencil." It doesn't work with our tokenizer, since it doesn't have spaces. You don't treat every character separately, either - 私 and は are their own thing, but えんぴつ means "pencil" and です is "to be."
"Eastern" languages need special tokenizing (and usually other treatment) when doing text analysis, mostly because they don't have spaces. They're collectively referred to as "CJK" languages, for Chinese, Japanese and Korean. It includes languages outside of those three, too, as long as they don't adhere to the "just make it lowercase and split on spaces" rules. You'll need to track down special tokenizers if you're working with those languages.
The scikit-learn package does a ton of stuff, some of which includes the above. We're going to start by playing with the CountVectorizer.
from sklearn.feature_extraction.text import CountVectorizer
count_vectorizer = CountVectorizer()
# .fit_transfer TOKENIZES and COUNTS
X = count_vectorizer.fit_transform(texts)
Let's take a look at what it found out!
X
Okay, that looks like trash and garbage. What's a "sparse array"??????
X.toarray()
If we put on our Computer Goggles we see that the first sentence has the first word 3 times, the second word 1 time, the third word 1 time, etc... But we can't read it, really. It would look nicer as a dataframe.
pd.DataFrame(X.toarray())
What do all of those numbers mean????
# Penny is a fish
# A fish is Penny
count_vectorizer.get_feature_names()
pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
So sentence #4 has "at" once, and the first sentence has "bought" once, and the last sentence has "the" three times. But hey, those are garbage words! They're cluttering up our dataframe! We need to add stopwords!
# We'll make a new vectorizer
count_vectorizer = CountVectorizer(stop_words='english')
#count_vectorizer = CountVectorizer(stop_words=['the', 'and'])
# .fit_transfer TOKENIZES and COUNTS
X = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())
pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
I still see meowed and meowing and fish and fishes - they seem the same, so let's lemmatize/stem them.
You can specify a preprocessor or a tokenizer when you're creating your CountVectorizer to do custom stuff on your words. Maybe we want to get rid of punctuation, lowercase things and split them on spaces (this is basically the default). preprocessor is supposed to return a string, so it's a little easier to work with.
# This is what our normal tokenizer looks like
def boring_tokenizer(str_input):
words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
return words
count_vectorizer = CountVectorizer(stop_words='english', tokenizer=boring_tokenizer)
X = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())
We're going to use one that features a stemmer - something that strips the endings off of words (or tries to, at least). This one is from nltk.
# https://tartarus.org/martin/PorterStemmer/def.txt
from nltk.stem.porter import PorterStemmer
porter_stemmer = PorterStemmer()
print(porter_stemmer.stem('fishes'))
print(porter_stemmer.stem('meowed'))
print(porter_stemmer.stem('oranges'))
print(porter_stemmer.stem('meowing'))
print(porter_stemmer.stem('orange'))
print(porter_stemmer.stem('go'))
print(porter_stemmer.stem('went'))
porter_stemmer = PorterStemmer()
def stemming_tokenizer(str_input):
words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
words = [porter_stemmer.stem(word) for word in words]
return words
count_vectorizer = CountVectorizer(stop_words='english', tokenizer=stemming_tokenizer)
X = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())
Now lets look at the new version of that dataframe.
pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
"Penny bought bright blue fishes.",
"Penny bought bright blue and orange fish.",
"The cat ate a fish at the store.",
"Penny went to the store. Penny ate a bug. Penny saw a fish.",
"It meowed once at the bug, it is still meowing at the bug and the fish",
"The cat is at the fish store. The cat is orange. The cat is meowing at the fish.",
"Penny is a fish"TF-IDF#
Part One: Term Frequency#
TF-IDF? What? It means term frequency inverse document frequency! It's the most important thing. Let's look at our list of phrases
- Penny bought bright blue fishes.
- Penny bought bright blue and orange fish.
- The cat ate a fish at the store.
- Penny went to the store. Penny ate a bug. Penny saw a fish.
- It meowed once at the fish, it is still meowing at the fish. It meowed at the bug and the fish.
- The cat is fat. The cat is orange. The cat is meowing at the fish.
- Penny is a fish
If we're searching for the word fish, which is the most helpful phrase?
pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
Probably the one where fish appears three times.
It meowed once at the fish, it is still meowing at the fish. It meowed at the bug and the fish.
But are all the others the same?
Penny is a fish.
Penny went to the store. Penny ate a bug. Penny saw a fish.
In the second one we spend less time talking about the fish. Think about a huge long document where they say your name once, versus a tweet where they say your name once. Which one are you more important in? Probably the tweet, since you take up a larger percentage of the text.
This is term frequency - taking into account how often a term shows up. We're going to take this into account by using the TfidfVectorizer in the same way we used the CountVectorizer.
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l1')
X = tfidf_vectorizer.fit_transform(texts)
pd.DataFrame(X.toarray(), columns=tfidf_vectorizer.get_feature_names())
Now our numbers have shifted a little bit. Instead of just being a count, it's the percentage of the words.
value = (number of times word appears in sentence) / (number of words in sentence)
After we remove the stopwords, the term fish is 50% of the words in Penny is a fish vs. 37.5% in It meowed once at the fish, it is still meowing at the fish. It meowed at the bug and the fish..
Note: We made it be the percentage of the words by passing in
norm="l1"- by default it's normally an L2 (Euclidean) norm, which is actually better, but I thought it would make more sense using the L1 - a.k.a. terms divided by words -norm.
So now when we search we'll get more relevant results because it takes into account whether half of our words are fish or 1% of millions upon millions of words is fish. But we aren't done yet!
Part Two: Inverse document frequency#
Let's say we're searching for "fish meow"
tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l1')
X = tfidf_vectorizer.fit_transform(texts)
df = pd.DataFrame(X.toarray(), columns=tfidf_vectorizer.get_feature_names())
df
What's the highest combined? for 'fish' and 'meow'?
# Just add the columns together
pd.DataFrame([df['fish'], df['meow'], df['fish'] + df['meow']], index=["fish", "meow", "fish + meow"]).T
Indices 4 and 6 (numbers 5 and 7) are tied - but meow never even appears in one of them!
It meowed once at the bug, it is still meowing at the bug and the fish
Penny is a fish
It seems like since fish shows up again and again it should be weighted a little less - not like it's a stopword, but just... it's kind of cliche to have it show up in the text, so we want to make it less important.
This is inverse term frequency - the more often a term shows up across all documents, the less important it is in our matrix.
# use_idf=True is default, but I'll leave it in
idf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=True, norm='l1')
X = idf_vectorizer.fit_transform(texts)
idf_df = pd.DataFrame(X.toarray(), columns=idf_vectorizer.get_feature_names())
idf_df
Let's take a look at our OLD values, then our NEW values, just for meow and fish.
# OLD dataframe
pd.DataFrame([df['fish'], df['meow'], df['fish'] + df['meow']], index=["fish", "meow", "fish + meow"]).T
# NEW dataframe
pd.DataFrame([idf_df['fish'], idf_df['meow'], idf_df['fish'] + idf_df['meow']], index=["fish", "meow", "fish + meow"]).T
Notice how 'meow' increased in value because it's an infrequent term, and fish dropped in value because it's so frequent.
That meowing one (index 4) has gone from 0.50 to 0.43, while Penny is a fish (index 6) has dropped to 0.40. Now hooray, the meowing one is going to show up earlier when searching for "fish meow" because fish shows up all of the time, so we want to ignore it a lil' bit.
But honestly I wasn't very impressed by that drop.
And this is why defaults are important: let's try changing it to norm='l2' (or just removing norm completely).
# use_idf=True is default, but I'll leave it in
l2_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=True)
X = l2_vectorizer.fit_transform(texts)
l2_df = pd.DataFrame(X.toarray(), columns=l2_vectorizer.get_feature_names())
l2_df
# normal TF-IDF dataframe
pd.DataFrame([idf_df['fish'], idf_df['meow'], idf_df['fish'] + idf_df['meow']], index=["fish", "meow", "fish + meow"]).T
# L2 norm TF-IDF dataframe
pd.DataFrame([l2_df['fish'], l2_df['meow'], l2_df['fish'] + l2_df['meow']], index=["fish", "meow", "fish + meow"]).T
LOOK AT HOW IMPORTANT MEOW IS. Meowing is out of this world important, because no one ever meows.
Who cares? Why do we need to know this?#
When someone dumps 100,000 documents on your desk in response to FOIA, you'll start to care! One of the reasons understanding TF-IDF is important is because of document similarity. By knowing what documents are similar you're able to find related documents and automatically group documents into clusters.
For example! Let's cluster these documents using K-Means clustering (check out this gif)
2 categories of documents#
# Initialize a vectorizer
vectorizer = TfidfVectorizer(use_idf=True, tokenizer=stemming_tokenizer, stop_words='english')
X = vectorizer.fit_transform(texts)
X
pd.DataFrame(X.toarray())
# KMeans clustering is a method of clustering.
from sklearn.cluster import KMeans
number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
km.labels_
texts
results = pd.DataFrame()
results['text'] = texts
results['category'] = km.labels_
results
4 categories of documents#
from sklearn.cluster import KMeans
number_of_clusters = 3
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
results = pd.DataFrame()
results['text'] = texts
results['category'] = km.labels_
results
Visualizing text similarity#
texts = ['Penny bought bright blue fishes.',
'Penny bought bright blue and orange bowl.',
'The cat ate a fish at the store.',
'Penny went to the store. Penny ate a bug. Penny saw a fish.',
'It meowed once at the bug, it is still meowing at the bug and the fish',
'The cat is at the fish store. The cat is orange. The cat is meowing at the fish.',
'Penny is a fish.',
'Penny Penny she loves fishes Penny Penny is no cat.',
'The store is closed now.',
'How old is that tree?',
'I do not eat fish I do not eat cats I only eat bugs']
# Initialize a vectorizer
vectorizer = TfidfVectorizer(use_idf=True, max_features=2, tokenizer=stemming_tokenizer, stop_words='english')
X = vectorizer.fit_transform(texts)
vectorizer.get_feature_names()
df = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())
df
ax = df.plot(kind='scatter', x='fish', y='penni', alpha=0.1, s=300)
ax.set_xlabel("Fish")
ax.set_ylabel("Penny")
from sklearn.cluster import KMeans
number_of_clusters = 3
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
df['category'] = km.labels_
df
color_list = ['r', 'b', 'g', 'y']
colors = [color_list[i] for i in df['category']]
ax = df.plot(kind='scatter', x='fish', y='penni', alpha=0.1, s=300, c=colors)
ax.set_xlabel("Fish")
ax.set_ylabel("Penny")
# Initialize a vectorizer
vectorizer = TfidfVectorizer(use_idf=True, max_features=3, tokenizer=stemming_tokenizer, stop_words='english')
X = vectorizer.fit_transform(texts)
vectorizer.get_feature_names()
df = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())
df
from sklearn.cluster import KMeans
number_of_clusters = 4
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
df['category'] = km.labels_
df['text'] = texts
df
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
def draw(ax, df):
color_list = ['r', 'b', 'g', 'y']
colors = [color_list[i] for i in df['category']]
marker_list = ['o', 'x', 'v', 'X']
markers = [marker_list[i] for i in df['category']]
ax.scatter(df['fish'], df['penni'], df['cat'], c=colors, s=100, alpha=0.5)
ax.set_xlabel('Fish')
ax.set_ylabel('Penni')
ax.set_zlabel('Cat')
chart_count_vert = 5
chart_count_horiz = 5
number_of_graphs = chart_count_vert * chart_count_horiz
fig = plt.figure(figsize=(3 * chart_count_horiz, 3 * chart_count_vert))
for i in range(number_of_graphs):
ax = fig.add_subplot(chart_count_horiz, chart_count_vert, i + 1, projection='3d', azim=(-360 / number_of_graphs) * i)
draw(ax, df)
