Word Embeddings: Comparing Word2Vec, GloVe, and fastText

Three Models, Three Philosophies

Word2Vec, GloVe, and fastText all aim to solve the same problem: representing words as dense numerical vectors. However, they approach this goal with fundamentally different strategies.

A great way to start is with a high-level overview. The following video provides a concise and clear introduction to the core idea behind each of these three influential models.

Now that you have a high-level mental map, let's explore each model's approach in more detail.

A Quick Recap: Word2Vec's Predictive, Local Approach

As we've covered, Word2Vec is a predictive model. Its training objective is to predict a word from its immediate neighbors (CBOW) or predict the neighbors from a central word (Skip-gram).

The key takeaway is that Word2Vec learns embeddings by processing a corpus sequentially and considering only local context windows. Its understanding of word relationships is built up implicitly from these local observations.

GloVe: A Global, Count-Based Approach

GloVe, which stands for Global Vectors for Word Representation, takes a different path. Instead of learning from local context windows one at a time, it is a count-based model that first aggregates global statistics from the entire corpus.

Training Approach:

1. Build a Co-occurrence Matrix: GloVe's first step is to construct a massive word-word co-occurrence matrix, which we can call $X$. An entry $X_{ij}$ in this matrix stores the number of times word j appears in the context of word i across the entire corpus. This is the "global" aspect.
2. Factorize the Matrix: The model then learns word vectors such that their dot product relates to the probability of their co-occurrence. The objective function aims to learn two sets of vectors, $w$ and $\tilde{w}$, such that for any two words i and j: $$w_i^T{\tilde{w}}_j+b_i+{\tilde{b}}_j\approx \log (X_{ij})$$ Here, $b_i$ and ${\tilde{b}}_j$ are bias terms. The model uses stochastic gradient descent to learn the vectors that best satisfy this relationship. The final embedding for a word is typically the sum of its corresponding vectors from both sets ($w_i+{\tilde{w}}_i$).

Given your background in data science, you can think of this as a specialized form of matrix factorization, akin to what's used in recommender systems, but adapted specifically for the statistical properties of word co-occurrences.

The following article gives a clear explanation of GloVe's mechanics.

Strengths of GloVe:

• Explicitly leverages global statistics, which can lead to high-quality embeddings.
• Often performs exceptionally well on word analogy tasks.
• Can train faster than Word2Vec since it iterates over the aggregated counts.

Weaknesses of GloVe:

• Requires significant memory to store the large co-occurrence matrix.
• Like Word2Vec, it cannot create vectors for words that were not in its training vocabulary (Out-of-Vocabulary or OOV words).

fastText: Embracing the Subword

While Word2Vec and GloVe treat words as atomic units, fastText, developed by Facebook AI Research, introduces a groundbreaking idea: representing a word as a bag of its character n-grams.

Training Approach:

The training process itself is nearly identical to Word2Vec's Skip-gram or CBOW models. The crucial difference is in how it represents a word. For example, with 3-grams (n=3), the word "apple" would be represented by the set of its character n-grams, plus the word itself:
{"<ap", "app", "ppl", "ple", "le>", "<apple>"}
(Special boundary symbols < and > are added to distinguish prefixes and suffixes).

The model then learns vectors for all these character n-grams. The final vector for the word "apple" is simply the sum or average of the vectors for its constituent n-grams.

This subword approach gives fastText its primary superpower. Let's watch a short clip that explains this idea very clearly.

Strengths of fastText:

• Handles OOV words: This is its key advantage. It can generate a vector for a word it has never seen before (e.g., "vectorizeable" or even a typo like "embddings") by summing the vectors of its n-grams.
• Excels with morphologically rich languages: In languages like German or Turkish where words have many forms (e.g., run, running, ran), fastText can capture the shared meaning because the word stems share common character n-grams.

Weaknesses of fastText:

• Increased memory and training time: The model needs to store vectors for a vast number of character n-grams, making it larger and potentially slower to train than Word2Vec.

Putting It All Together

We've examined the unique philosophy behind each model. To solidify your understanding, let's bring it all together in a comparative summary.

The following article offers a useful wrap-up, including a section on how to choose the right model for a given task.

Here is a summary table that contrasts the three models:

Conclusion

In this lesson, you've broadened your understanding of static word embeddings beyond Word2Vec. You can now compare and contrast the three major players of that era.

Here are the key takeaways:

• Word2Vec is a predictive model that learns from local context windows.
• GloVe is a count-based model that learns by factorizing a global word-word co-occurrence matrix.
• fastText is a predictive model that enhances Word2Vec by representing words as a bag of character n-grams, enabling it to handle out-of-vocabulary words.

A crucial point to carry forward is that all three of these models produce static embeddings. The word "bank" has the same vector in "river bank" as it does in "investment bank." This is a significant limitation.

This sets the stage perfectly for our next module, where we will begin our journey toward contextual embeddings. In the next lesson, we will explore Recurrent Neural Networks (RNNs), the first major architecture designed to process sequential data like text and a foundational step toward models that can understand that the meaning of a word depends on its context.