Word2Vec Architectures: Skip-gram and CBOW

The Core Idea: Learning Meaning from Context

The philosophy behind Word2Vec is elegantly summarized by the linguistic principle: "You shall know a word by the company it keeps." Instead of manually defining the meaning of a word, we can infer it by observing the words that typically surround it in vast amounts of text.

Word2Vec cleverly operationalizes this idea by training a simple neural network on a "pretext" or "fake" task. The goal isn't to build a state-of-the-art language model, but to force the network to learn good word embeddings as a side effect of solving this task. The learned weights of one of the network's layers are, in fact, the word embeddings we seek.

The following video provides an excellent introduction to this concept, framing the process as solving a "fake problem" to learn embeddings as a "side effect."

Generating Training Data with a Sliding Window

To learn from context, we first need to generate training data. Word2Vec does this by sliding a "window" of a fixed size across a massive corpus of text (like all of Wikipedia). As the window moves, it creates training samples, each consisting of a central "target" word and its surrounding "context" words.

For example, consider the sentence: "The quick brown fox jumps over the lazy dog."

If our window size is 5 (a center word with two words before and two after), and the center word is "fox," our training data point would be:

• Target: fox
• Context: (The, quick, brown, jumps)

This process is repeated for every word in the corpus, generating millions of training samples. The "Illustrated Word2Vec" article by Jay Alammar provides a superb visual explanation of this process.

Once we have this dataset of (context, target) pairs, we can train a model. Word2Vec proposes two main architectures for this, which essentially flip the problem on its head.

Let's examine each one.

1. Continuous Bag-of-Words (CBOW)

The CBOW model takes the surrounding context words as input and tries to predict the single target word in the center. The name "Bag-of-Words" comes from the fact that the model treats the context words as an unordered collection, typically by averaging their corresponding vectors.

The task: Given (The, quick, ___, jumps, over), predict fox.

The architecture:

1. Input Layer: The context words (e.g., "The", "quick", "jumps", "over") are one-hot encoded.
2. Projection (Hidden) Layer: The one-hot vectors are used to look up their corresponding embedding vectors from an embedding matrix. These vectors are then averaged to create a single context vector.
3. Output Layer: A final dense layer with a softmax activation function produces a probability distribution over the entire vocabulary. The goal is to maximize the probability of the true target word ("fox").

The codebasics video you watched earlier provides a great walkthrough of the CBOW architecture.

2. Skip-gram

The Skip-gram model does the exact opposite of CBOW. It takes the central target word as input and tries to predict the surrounding context words.

The task: Given fox, predict (The, quick, jumps, over).

The architecture:

1. Input Layer: The single target word (e.g., "fox") is one-hot encoded.
2. Projection (Hidden) Layer: The embedding for the input word is retrieved from the embedding matrix.
3. Output Layer: The model has multiple output layers, each with a softmax activation, which independently try to predict each of the words in the context window.

For a single training step with the target "fox" and a context window of size 2 (one word before, one after), Skip-gram would generate two separate training examples: (fox, brown) and (fox, jumps). This makes Skip-gram more computationally intensive than CBOW but allows it to learn better representations for infrequent words.

Let's look at a more detailed visualization of the Skip-gram architecture.

The following video provides another clear, concise explanation of both CBOW and Skip-gram.

The Training Challenge and a Clever Solution: Negative Sampling

There's a major practical challenge with the architectures described so far. The final softmax layer in both CBOW and Skip-gram is extremely expensive to compute. For every training sample, the model would need to calculate a probability for every single word in the vocabulary (which can be millions of words) just to normalize the output.

To make training feasible, the Word2Vec authors introduced a technique called Negative Sampling. Instead of framing the task as a massive multi-class classification problem ("predict the correct word"), it is reframed as a much simpler binary classification problem:

• Given a pair of words, (input_word, output_word), predict whether output_word is a real context word for input_word (label = 1) or just a random word from the vocabulary (label = 0).

For each real context pair (a "positive sample"), we generate several "negative samples" by pairing the input word with randomly chosen words. The model's job is then to learn to distinguish the true pairs from the fake ones. This is vastly more efficient because the model only needs to update the weights for the few words in each training step (the positive sample and a handful of negative ones), rather than for the entire vocabulary.

The "Illustrated Word2Vec" article provides a fantastic, step-by-step walkthrough of this crucial optimization.

Conclusion

In this lesson, we've unpacked the "how" behind Word2Vec. You now understand the mechanisms that produce the dense vector representations we discussed in the previous lesson.

Here are the key takeaways:

• Learning from Context: Word2Vec learns embeddings by training a neural network on a pretext task of predicting words from their context, based on the principle of distributional semantics.
• Two Architectures:
  • CBOW (Continuous Bag-of-Words): Predicts a central target word from its surrounding context words. It's faster and works well for frequent words.
  • Skip-gram: Predicts surrounding context words from a central target word. It's slower but generally produces better embeddings, especially for rare words.
• Efficient Training: The computationally expensive softmax over the entire vocabulary is replaced by Negative Sampling, which reframes the task as a more efficient binary classification problem of distinguishing true context pairs from random negative ones.

You've now moved from understanding why embeddings are useful to knowing how they are learned. This is a crucial step towards demystifying the models you use in your work.

In our next lesson, we will explore the fascinating results of this process. We'll see how the learned vector space encodes semantic relationships, allowing us to perform a kind of "vector arithmetic" with words to solve analogies like king - man + woman ≈ queen.