Words to Vectors
Engineers working at Google were somewhat surprised to discover that you can do algebra with words. The way to do this practically is to have the computer learn which words appear together (e.g. king and queen), and then map them in space. You can then do arithmetic with words (e.g. subtract man from king). And that is exactly what you are going to be doing today.
From Words to Vectors
You may have already been using vectors without knowing it with latitude and longitude. For example, the Empire State Building in New York City is at [40.748817°, -73.985428°], a two-dimensional vector. We can add the third dimension of height: the top of the Empire State Building is at 443.2 meters from street to antenna tip. If the fourth dimension is time, we can use midnight on January 1st, 2030, which is 189.3474 × 107 seconds since January 1, 1970 (Unix time). So, if you had a time machine, and wanted to meet “at the top of the Empire State Building at midnight on January 1, 2030” you might set it for the four-dimensional space-time vector of [40.748817, -73.985428, 443.2, 189.3474].
We can also use vectors for words. For example, the vector for apples might be [0.21, -0.44, 0.73, 0.08] and the vector for oranges might be [0.18, -0.39, 0.69, 0.12]. We don’t pull those numbers out of a hat, though. We feed the computer large amounts of text, and it learns what the vectors for words are through trial and error. Let’s do it.
Install the Gensim library using pip.
pip install gensimNow, import the Word2Vec class from the Gensim models package.
from gensim.models import Word2VecLet’s try it out with the sentences “you’re adding apples and oranges” and “this is far out”, all lowercase and separated out into individual words.
sentences = [
["you're", "adding", "apples", "and", "oranges"],
["this", "is", "far", "out"],
]Now train the model. It will read the sentences and guess what the vectors should be.
model = Word2Vec(
sentences=sentences, vector_size=4, min_count=1
)Now, get the vectors for both “apples” and “oranges”:
apples_vector = model.wv["apples"]
print("Apples vector:", apples_vector)
oranges_vector = model.wv["oranges"]
print("Oranges vector:", oranges_vector)In the output, we now see the vectors for “apples” and “oranges”:
Apples vector: [ 0.15877226 -0.08513415 -0.02366003 0.14421433]
Oranges vector: [ 0.07191449 0.02479684 -0.20713037 -0.23622045]Add Apples and Oranges
It’s time for the big one: adding apples and oranges. What was missing from our example before was tons of text to train the model on. Google engineers casually mention billions of words of text. In fact, their breakthrough was creating a training method that could work on billions of words without taking forever. They have huge server farms though, and we have laptops. So, we are going to use a readymade model for this experiment.
To start, import the Gensim downloader:
>>> import gensim.downloaderLoad Stanford’s GloVe Wiki Gigaword 50 corpus.
>>> model = gensim.downloader.load("glove-wiki-gigaword-50")Drum roll…. let us add apples and oranges. Call the most_similar method of the loaded model on the positive list of things to be added: ["apples", "oranges"], and store it in the result variable.
>>> result = model.most_similar(positive=["apples", "oranges"])
>>> print(result)Lo and behold, you can add "apples" and "oranges", and the result is… 'peaches'!
[
('peaches', 0.8826718330383301),
('cherries', 0.8604050874710083),
('strawberries', 0.8487799167633057),
('apricots', 0.8453282117843628),
('mangoes', 0.8449215292930603),
('melons', 0.8295340538024902),
('plums', 0.8263391852378845),
('pineapples', 0.8190645575523376),
('lemons', 0.8119077682495117),
('bananas', 0.8056995868682861)
]What’s going on here is 'peaches' is most similar to the sum of “apples” plus “oranges”, with a similarity score of 0.88, 'cherries' is next at 0.86, and then 'strawberries' at 0.85, etc.
Algebra with words:
Now the fun part starts. As promised, we are going to be doing math with words. A famous example is from the 2013 paper. The idea is we take the word "king", subtract the word "man", and add the word "woman". This means something like, “what is an emasculated, feminized king?”
>>> result = model.most_similar(positive=["king", "woman"], negative=["man"])
>>> print(result)The answer is 'queen'. The word closest to an emasculated, feminized king is “queen”, with a similarity score of 0.85.
[('queen', 0.8523604273796082), ('throne', 0.7664334177970886), ('prince', 0.7592144012451172), ('daughter', 0.7473883628845215), ('elizabeth', 0.7460220456123352), ('princess', 0.7424570322036743), ('kingdom', 0.7337411642074585), ('monarch', 0.721449077129364), ('eldest', 0.7184861898422241), ('widow', 0.7099431157112122)]