Comparing documents across languages with Universal Sentence Encoding and Tensorflow#
What do we do when we have terabytes of documents scattered across multiple languages? Well, if we find one document that's interesting, we might want to ask the computer to anything that's similar to it. If we ask especially politely, we can have it find similar documents even in a different language.
I found out about this technique based on a writeup of Quartz's analysis of the Luanda Leaks. I recommend giving it a read-through before you go through here, just for a bit of context.
Note: I talk about documents a lot in this section, but what we're really interested in is sentences. When we get to the next section - how to apply these techniques to large datasets - the difference will become more clear.
Let's say we have a handful of sentences.
import pandas as pd
sentences = [
"Molly ate a fish",
"Jen consumed a carp",
"I would like to sell you a house",
"Я пытаюсь купить дачу", # I'm trying to buy a summer home
"J'aimerais vous louer un grand appartement", # I would like to rent a large apartment to you
"This is a wonderful investment opportunity",
"Это прекрасная возможность для инвестиций", # investment opportunity
"C'est une merveilleuse opportunité d'investissement", # investment opportunity
"これは素晴らしい投資機会です", # investment opportunity
"野球はあなたが思うよりも面白いことがあります", # baseball can be more interesting than you think
"Baseball can be interesting than you'd think"
]
I used Google Translate to mix and match between languages - some Russian, some Japanese, some French - to varying degrees of similarity. Some are exactly the same (investment opportunities), while others are only roughly about the same topic (renting or buying houses/apartments).
Without spending time going through them one-by-one ourselves, how can we find sentences that are similar to one another?
Old method: Counting words#
Traditionally, document similarity is based on the words two documents have in common.
First, we'll count the number of times each word appears.
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(binary=True)
matrix = vectorizer.fit_transform(sentences)
counts = pd.DataFrame(
matrix.toarray(),
index=sentences,
columns=vectorizer.get_feature_names())
counts.head()
Then we'll see how many words each sentence has in common with each other sentence. The more words two sentences have in common, the higher their similarity should be.
from sklearn.metrics.pairwise import cosine_similarity
# Compute the similarities using the word counts
similarities = cosine_similarity(matrix)
# Make a fancy colored dataframe about it
pd.DataFrame(similarities,
index=sentences,
columns=sentences) \
.style \
.background_gradient(axis=None)
Pretty boring, right? These sentences share almost no words (ignoring things like a or the), so the only two sentences that are actually marked as similar are...
Baseball can be interesting than you'd thinkI would like to sell you a house
...because they both contain the word you! While it's useless, it isn't unexpected. These sentences are all in different languages, how in the world are we supposed to judge whether they're similar or not?
New method: Universal sentence encoder#
Once upon a time we talked about word embeddings, which are ways for each word to have multiple dimensions of meaning. "cat" and "lion" might both be catlike, while "lion" and "wolf" are both wild.
Imagine a graph that looks like this, but with three hundred dimensions:
To find words that are similar, you just find ones that are close to each other in that 300-dimension space: a certain amount about cats, a certain amount wild, a certain amount edible, a certain amount red, etc etc etc. Notice in the chart above, shoe is far far off to the left: that means it isn't very similar to those other four words! If you haven't seen it yet, it's a great idea to go read our word embeddings page for more details.
Researchers took this idea of word embeddings and used some fun computer magic to take it one step further: they learned to apply it across different languages!
We aren't talking just strict translation! While yes, cat and gato and 猫 all translate to the same word, multi-language word embeddings mean a lot more. A sentence that talks about meowing can be marked as similar to one that talks about gatos, even though the words aren't exact translation matches, just because both of those words are cat-related!
The Multilingual Universal Sentence Encoder is our new best friend. Using it along with Tensorflow, we'll be able to match up our similarly sentences, even if they're in completely different languages.
Big thanks to Jeremy Merrill's tensorflow v1 example Asia spa, even though I can't agree with his choice in bagels
And hey, 300 dimensions? Forget about that, let's upgrade to 512.
The code#
If you need to install tensorflow or its associated packages, uncomment and run the next line. Otherwise we're pretty much good to go!
# !pip install tensorflow tensorflow_hub tensorflow_text
# Import tensorflow and friends
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text
We'll start by loading the Multilingual Universal Sentence Encoder. We're using version 3, which is super user-friendly.
I believe this requires that we're Tensorflow v2, but don't quote me on that.
# Load the Multilingual Universal Sentence Encoder, v3
embed = hub.load("https://tfhub.dev/google/universal-sentence-encoder-multilingual/3")
We can now use this embed to create our multilingual sentence embeddings. Congratulations!
What's it look like when we run an encoding? Let's find the 512 dimensions of knowledge about bagels.
embed("the only kind of bagel is everything")
Fun, right? So now we're going to feed all of our sentences into the encoder. Each sentences will get its own 512-dimensional representation, and then we'll use that to see which ones are close to each other.
# Generate embeddings for each sentence
embeddings = embed(sentences)
from sklearn.metrics.pairwise import cosine_similarity
# Compute similarities exactly the same as we did before!
similarities = cosine_similarity(embeddings)
# Turn into a dataframe
pd.DataFrame(similarities,
index=sentences,
columns=sentences) \
.style \
.background_gradient(axis=None)
Magic, right?
The ones about housing are all grouped together, investment opportunities are marked as similar, and baseball as well. You'll notice it (somewhat obviously) even works within the same language - Jen consumed a carp and Molly ate a fish are both similar.
While this is fun conceptually and all, next up we'll see how to put this into production use!