Word Embeddings: Word2Vec and GloVe

Hello! Welcome to your next lesson in the Natural Language Processing module.

In the last lesson, we built text classification models using TF-IDF vectors. While powerful, that approach has a significant limitation: it treats words as isolated tokens. For a TF-IDF-based model, the words "king" and "queen" are just as different from each other as "king" and "keyboard." It has no understanding of semantic meaning or relationships.

Today, we'll tackle this problem head-on. Our learning outcome is to explain word embeddings and use pre-trained Word2Vec/GloVe vectors. We will explore how to represent words as dense numerical vectors that capture their meaning and context. This is a foundational concept in modern NLP and our bridge into the world of deep learning for text.

1. Beyond Word Counts: The Need for Dense Representations

Before we dive into specific models, let's solidify our understanding of what word embeddings are and why they are so revolutionary. The traditional methods you've seen, like Bag-of-Words and one-hot encoding, suffer from two major problems:

Sparsity and High Dimensionality: The vectors are mostly zeros and their length equals the vocabulary size, which can be massive. This is computationally inefficient.
Lack of Semantic Meaning: These representations don't capture relationships. The vector for "cat" is mathematically equidistant from "dog" and "airplane."

Word embeddings solve this by creating dense, low-dimensional vectors where semantic similarity is encoded in the geometry of the vector space.

Word Embeddings: From Word2Vec to GloVe

To start, let's read an excellent article by Michael Brenndoerfer that introduces the concept of word embeddings and contrasts them with traditional methods. This will set the stage for the rest of the lesson.

Please read the following sections: 'Word Embeddings' and 'The Problem with Traditional Representations': Focus on understanding the limitations of one-hot encoding and the 'curse of dimensionality'. 'Word Embedding' (definition box): This provides a concise definition that you should internalize. The core idea that makes this possible is the distributional hypothesis: words that appear in similar contexts tend to have similar meanings.

The key is that we can learn these representations automatically from vast amounts of text. Let's look at two of the most influential algorithms for doing this: Word2Vec and GloVe.

2. Word2Vec: Learning from Local Context

Word2Vec, developed by researchers at Google, was a breakthrough in learning high-quality word embeddings. Its core idea is to train a simple neural network on a "fake" predictive task. As a side effect of learning to solve this task, the network's internal weights become the word embeddings we desire.

Let's watch a video that explains the intuition behind this process.

What is Word2Vec? A Simple Explanation | Deep Learning Tutorial 41 (Tensorflow, Keras & Python)

This video from the 'codebasics' channel provides a simple, intuitive explanation of Word2Vec's core concepts.

Watch the first two parts of the video (00:00 - 07:37): The Core Idea (00:00 - 04:21): Pay attention to the famous analogy King - Man + Woman = Queen. This demonstrates the power of representing words as vectors that capture relationships. The 'Fake Problem' (04:21 - 07:37): This is the key insight. Understand how the goal is not actually to predict missing words, but to learn the embeddings as a 'side effect' of this task.

As the video explained, the meaning of a word can be inferred from its surrounding words (its "context"). Word2Vec formalizes this with two main model architectures:

Continuous Bag-of-Words (CBOW): Predicts a target word based on its context words.
Skip-gram: Predicts the context words based on a single target word. (This is generally preferred as it works better for infrequent words).

CBOW vs. Skip-gram Architectures — This diagram illustrates the two main architectures of Word2Vec. CBOW (left) predicts the center word from the context. Skip-gram (right) does the inverse, predicting context words from the center word.

The codebasics video continues to explain these architectures in more detail, showing the simple neural network structure used. The crucial part is that the learned weight matrix between the input and hidden layer is extracted and used as our final word embeddings.

3. GloVe: Learning from Global Statistics

While Word2Vec learns by iterating through local context windows, GloVe (Global Vectors for Word Representation), from Stanford researchers, takes a different approach. It first aggregates global co-occurrence statistics across the entire corpus and then learns embeddings from this data.

The key insight of GloVe is that ratios of co-occurrence probabilities can reveal interesting semantic relationships. For example:

The ratio of P("solid" | "ice") to P("solid" | "steam") will be very high.
The ratio of P("gas" | "ice") to P("gas" | "steam") will be very low.
The ratio of P("water" | "ice") to P("water" | "steam") will be close to 1.

GloVe is designed to learn word vectors such that their dot products relate directly to the logarithm of their co-occurrence counts.

Introduction to NLP | GloVe Model Explained

Let's watch a video from 'Normalized Nerd' that explains the intuition behind GloVe and how it differs from Word2Vec.

Please watch the following segments (00:56 - 08:58): GloVe vs. Word2Vec (00:56 - 02:30): Understand the fundamental difference: GloVe uses global statistics, while Word2Vec uses local context. Co-occurrence Matrix (02:30 - 04:43): See how the global statistics are first captured in a large matrix. Power of Ratios (04:43 - 08:58): This is the core mathematical insight behind GloVe. Focus on the 'ice' vs. 'steam' example to understand why ratios are more informative than raw probabilities.

A Quick Comparison

To solidify your understanding, let's briefly compare the two approaches.

Word Embeddings: From Word2Vec to GloVe

The article by Michael Brenndoerfer provides a concise summary comparing the two models.

Please read the sections 'Comparing Word2Vec and GloVe' and 'Summary'. This will help you understand the trade-offs and consolidate the key takeaways.

In practice, both Word2Vec and GloVe produce high-quality embeddings. The choice often depends on the specific task, but more importantly, it's rare to train these models from scratch. Instead, we use pre-trained models.

4. Using Pre-trained Word Embeddings in Python

Training your own embeddings requires a massive text corpus (billions of words) and significant computational resources. Fortunately, researchers have trained models on large datasets (like Wikipedia and Google News) and made them publicly available.

We will use the gensim library, a popular and powerful toolkit for topic modeling and vector space semantics in Python, to load and work with these pre-trained vectors.

Let's walk through a practical example.

Advanced - Word Embeddings (Python)

This tutorial from UBC COMET provides a clear, step-by-step guide to loading and using a pre-trained Word2Vec model with gensim.

Please follow the code in 'Exercise #3: Using word2vec with 150 English Novels'. Focus on these parts: Loading the Model: Find the line model = gensim.models.KeyedVectors.load_word2vec_format(...). This is how you load a pre-trained model file. Getting a Vector: The model.get_vector('whale') command shows you how to retrieve the dense vector for a specific word. Vector-Space Operations: This is the most exciting part. Read through and understand the code for: Similarity: model.similarity('word1', 'word2') Finding Neighbors: model.most_similar('word') Analogy: model.most_similar(positive=['woman', 'king'], negative=['man']) You have a solid Python background, so I encourage you to set up a new Python environment, install gensim (pip install gensim), download a pre-trained model (like the popular GloVe vectors from Stanford's website), and run these commands yourself.

The process for loading GloVe vectors is very similar. The Normalized Nerd video shows a quick demonstration of this at the end (from 19:43 onwards), confirming that gensim can handle both formats.

Word Embeddings Visualization — This image visualizes the vector arithmetic behind the 'king - man + woman = queen' analogy. The learned embeddings place words in the vector space such that these semantic relationships hold true.

Conclusion

You have now made a significant leap from treating words as simple tokens to understanding them as rich, semantic vectors. This ability to capture meaning is the foundation for most modern NLP models.

Key Takeaways:

Word embeddings are dense, low-dimensional vector representations of words that capture semantic meaning.
The distributional hypothesis ("a word is known by the company it keeps") is the guiding principle for learning embeddings.
Word2Vec learns embeddings from local context windows, using architectures like CBOW and Skip-gram.
GloVe learns embeddings from a global co-occurrence matrix, focusing on ratios of co-occurrence probabilities.
In practice, we often use pre-trained embeddings loaded with libraries like gensim to perform powerful semantic operations like finding word similarity and solving analogies.

Preview of the Next Lesson:
Word2Vec and GloVe embeddings are powerful, but they are static. The vector for the word "bank" is the same whether it appears in "river bank" or "investment bank." How can a model understand the meaning of a word in the context of a specific sentence? In the next lesson, we will explore Recurrent Neural Networks (RNNs) and LSTMs, architectures designed specifically to process sequences of vectors (like a sequence of word embeddings) and maintain a memory of what they've seen.