Conceptual document similarity with word embeddings#
There are a lot of different ways to compare the similarity of words and documents. Sometimes it's technical, sometimes it's a little more theoretical. In this section we'll look at comparing the concepts of multiple documents instead of just comparing the words.
The issue#
Computers are very, very, very particular about what counts as "the same." For example, cat is the same as cat but the capitalized version of cat is not the same as the small version.
print('Is cat the same as cat?', "cat" == "cat")
print('Is CAT the same as cat?', "CAT" == "cat")
If we're looking at similarity just using the letters in the words, we can do things like rearrange letters, or use edit distance for partial matches or all sorts of things in order to make "better" comparisons.
This pickiness ends up being a bit of a pain when comparing documents to se how similar they are to one another. No matter what we do, no amount of counting words is going to get us the concepts in the piece, or the idea of what it's really about.
Let's start by taking a look at "normal" document similarity, then compare that approach with using word embeddings on the exact same sentences. Here are the sentences we'll be using.
sentences = [
'Molly ate a donut',
'Molly ate a fish',
'Jen consumed a carp',
'Lenny fears the lions'
]
print('\n'.join(sentences))
Which pair do you think is the most similar? Two of them are Molly eating something, while two of them are women eating fish. Lenny is... definitely an outlier. While you weigh the options, let's get analyzing!
Word counting#
If you read our piece on n-gram document similarity, you know that there's a lot that goes into whether two documents are similar or not. And if you didn't, no big deal! It totally doesn't matter.
To judge similarity between these sentences, we're going to use a TfidfVectorizer from scikit-learn. Less common words are stressed, more common words are more important, and words in long sentences mean less than words in short sentences. We're counting words in a fancy way.
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(binary=True)
matrix = vectorizer.fit_transform(sentences)
counts = pd.DataFrame(
matrix.toarray(),
index=sentences,
columns=vectorizer.get_feature_names())
counts
We'll be measuring similarity via cosine similarity, a standard measure of similarity in natural language processing. It's similar to how we might look at a graph with points at (0,0) and (2,3) and measure the distance between them - just a bit more complicated.
from sklearn.metrics.pairwise import cosine_similarity
# Compute the similarities using the word counts
similarities = cosine_similarity(matrix)
# Make a fancy colored dataframe about it
pd.DataFrame(similarities,
index=sentences,
columns=sentences) \
.style \
.background_gradient(axis=None)
Document similarity is on a scale of zero to one, with zero being completely dissimilar and one being an exact match. Each sentence has a 1 when compared to itself - they're totally equal!
- "Molly ate a donut" and "Molly ate a fish" are both pretty similar - over half - since there's only one word that's different between the two.
- "Jen consumed a carp" only has the nigh-useless "a" in common with them, so it has a similarity score of 0 to both of the others.
- Lenny's sentence also has no shared words with anything else. Nor any topics in common, although it doesn't matter right now.
In our brains, though, consumed means just about the same thing as ate. And a carp is a kind of fish, right? If only there were some way of teaching a computer the meaning behind words!
Word embeddings#
Word embeddings are a step up from just counting words. Word embeddings give words meaning to computers, teaching it that puppies are kind of like kittens, kittens are like cats, and shoes are very very different from all of those animals.
We're going to be using the spaCy word embeddings. Each word comes with a 300-dimension vector that expresses things like how catlike the word is, whether you can wear it, if it's something people do during a basketball game (not those exactly, but the same idea). Think of it like 300 different scores for each word, all in different categories.
If you're running this notebook yourself, it'll take a while to load it in. It's a lot of data!
import spacy
nlp = spacy.load("en_core_web_md")
For example, let's check out the 300 dimensions of facts and feelings that spaCy knows about the word cat.
nlp('cat').vector
We have absolutely no idea what it means, but sure, okay! If you don't trust it means something, go pop back to the original word embeddings section where we make some nice charts and graphs about puppies and kittens. They'll hopefully convince you!
In the same way that each word has a 300-dimension vector, entire sentences are just combinations of the words inside. We can feed spaCy a sentence and it'll spit out another 300 numbers, just like it did for cat.
nlp('Some people have never eaten a taco').vector
Again: no idea what it means, but we'll trust it works (and we'll see later on!).
In order to find the similarity of each of our sentences, we'll need to convert them each into vectors.
# We aren't printing this because it's 3 * 300 = 900 numbers
vectors = [nlp(sentence).vector for sentence in sentences]
# Print out some notes about it
print("We have", len(vectors), "different vectors")
print("And the first one has", len(vectors[0]), "measurements")
print("And the second one has", len(vectors[1]), "measurements")
print("And the third one has", len(vectors[2]), "measurements")
print("And the fourth one has", len(vectors[3]), "measurements")
It might be useful to compare these 300 measurements-per-sentence to what we were doing before. If we look back to when we were doing similarity by counting words, we only had eleven measurements for each sentence: one count for every unique word.
counts
Those extra 289 dimensions that word embeddings bring are packed with concepts instead of just word comparisons. Fishiness, eatinginess, catlikeness, it's all in there! (kind of, maybe, somewhat)
We'll use the same similarity measure as before, cosine similarity. It takes our numeric description of each sentence and checks the distances between them.
# Compute similarities
similarities = cosine_similarity(vectors)
# Turn into a dataframe
pd.DataFrame(similarities,
index=sentences,
columns=sentences) \
.style \
.background_gradient(axis=None)
And there we go! The first thing to notice is that none of these sentences are totally dissimilar. Last time we had multiple zeroes - this time there are none at all, not even with Lenny's weird adventures.
Next up and most importantly: "Molly ate a donut" and "Molly ate a fish" are still very similar with a choice in the 80's, but it turns out "Molly ate a fish" is also almost just as similar to "Jen consumed a carp"!
0.86394eating a donut vs. eating a fish0.86014eating a fish vs. consuming a carp
Thanks to spaCy's word embeddings understanding the concepts behind the words, the computer was able to measure the similarity between "ate" and "consumed," along with "carp" and "fish." As a result we got magically high scores despite the important words not matching exactly!
We might also note that the "fish" and "carp" sentences are slightly more similar to Lenny's lions than the donut sentence. Might it be because those three involve animals?
Gotchas#
While this is definitely exciting, using word embeddings for similarity isn't without its limits. The biggest issue is what we think is similar compared to what the computer thinks is similar. Because of the magic of word embeddings, we can't exactly measure what the computer knows!
# Here are our sentences
sentences = [
'Veronica hates mustard',
'Veronica loves ketchup',
'Joseph hates ketchup',
]
# Turn into vectors
vectors = [nlp(sentence).vector for sentence in sentences]
# Compute similarities
similarities = cosine_similarity(vectors)
# Turn into a dataframe
pd.DataFrame(similarities,
index=sentences,
columns=sentences) \
.style \
.background_gradient(axis=None)
So what's more similar, two people hating condiments or two people having emotional feelings about ketchup? This is a terrible example, but you'll survive until I figure out a better one.
Review#
In this section, we looked at how word embeddings can allow more in-depth comparisons between texts than just plain word counting. Word embeddings allow the computer to understand nuance and conceptual similarity as opposed to just word-by-word counting.
There are shortcomings to both word counting and word embeddings, but depending on whether you're looking for exact matches or a more conceptual pairing, either one can be the correct choice.
Discussion topics#
I'm looking for news websites that republish works without attribution, giving them slight edits. Would I search for them using word counts or word embeddings? What would be the benefit compared to doing this versus reading them manually?
If I wanted to compare the platforms of two politicians, should I use word counts or word embeddings? What would be the benefit compared to doing this versus reading them manually?
If I wanted to compare the platforms of hundreds of local races, should I use word counts or word embeddings? What would be the benefit compared to doing this versus reading them manually? What about using something like topic modeling instead?