Word Embeddings: Representing Meaning as Vectors

Hello! Welcome to the first lesson in our new module, "LLM and Transformer Fundamentals."

In the previous module, we treated Large Language Models (LLMs) largely as black boxes, focusing on the systems and economics surrounding their use—cost calculation, optimization, and intelligent routing. As we concluded, the next logical step is to start opening that black box.

Today, we begin that journey by addressing the most fundamental question: how do we translate human language into a format that a machine can understand and reason with? This lesson will cover the learning outcome: Explain how word embeddings represent semantic meaning as dense vectors. This is the first and most critical step in enabling models to process and generate text.

1. From Words to Numbers: The Challenge

Machine learning models operate on numbers, not words. So, our first task is to devise a numerical representation for every word in our vocabulary. Let's consider a couple of straightforward approaches and see why they fall short.

To understand the limitations of simple numerical representations, please watch the following video segment.

As the video explains, the two simple methods have critical flaws:

• Token IDs (Integer Encoding): Assigning good = 6, great = 21, and bad = 22 imposes an arbitrary and misleading mathematical relationship. The model might incorrectly infer that "great" and "bad" are similar because their numerical values are close, while "good" is distant. This method fails because it introduces an ordinal relationship where none exists.

• One-Hot Encoding: This method represents each word as a vector the size of the entire vocabulary, with a 1 at the word's unique index and 0s everywhere else.

  • Pro: It solves the ordinality problem. Every word vector is equidistant from every other.
  • Cons:
    1. Sparsity and Inefficiency: For a typical vocabulary of 50,000 words, each word becomes a 50,000-dimensional vector that is almost entirely zeros. This is extremely inefficient in terms of memory and computation.
    2. No Semantic Information: The vectors are orthogonal. The dot product of the vectors for "king" and "queen" is zero, just like the dot product for "king" and "chainsaw". This representation tells the model nothing about the relationships between words.

We need a representation that is both efficient and captures semantic meaning.

2. The Solution: Dense Vector Embeddings

The breakthrough came from representing words not as sparse, high-dimensional vectors, but as dense, lower-dimensional vectors, which we call word embeddings.

This approach is built on the distributional hypothesis, a core concept in linguistics often summarized as: "You shall know a word by the company it keeps" (J.R. Firth, 1957). The idea is that words that appear in similar contexts (i.e., surrounded by similar words) tend to have similar meanings.

To get a formal definition and contrast it with sparse vectors, let's turn to a short article.

A word embedding is a learned, dense vector of floating-point numbers that represents a word.

• Dense: Unlike one-hot vectors, most of the numbers are non-zero.
• Lower-dimensional: The vectors typically have a few hundred dimensions (e.g., 300, 768), which is much more manageable than the 50,000+ dimensions of one-hot vectors.
• Learned: These vectors are not manually designed. They are the result of training a model on a vast amount of text. The model learns to place words with similar meanings close to each other in a multi-dimensional "semantic space."

The image below provides a great visualization of this concept. Words are mapped to dense vectors, which can then be projected into a 2D space for visualization. Notice how semantically similar words like "cat" and "kitten" or "man" and "woman" are clustered together.

3. The Power of Embeddings: Capturing Relationships

The truly remarkable property of word embeddings is that the spatial relationships within the vector space capture complex semantic relationships. This allows us to perform "vector arithmetic" with words.

The most famous example is:

This demonstrates that the embedding space has learned abstract concepts like gender. The vector difference between "king" and "man" represents the concept of "royalty," which can then be applied to "woman" to find "queen."

Let's watch a video that explains this property intuitively.

This property isn't limited to gender. It extends to many other relationships, such as country-capital, verb tenses, and more.

4. How Are Embeddings Learned? An Intuitive Overview

So, how do we create these magical vectors? We don't program the relationships manually. We set up a clever, self-supervised learning task and let a neural network learn the vectors for us. Word2Vec is the classic model that popularized this approach.

The core idea is to train a model to do a "fake" task. For the Skip-gram version of Word2Vec, the task is: given a target word, predict its neighboring context words.

The StatQuest channel provides one of the best intuitive explanations of this process.

To summarize the key insight from the video:

1. Self-Supervision: We take a massive text corpus (like all of Wikipedia) and generate training examples automatically. For a sentence like "...the quick brown fox jumps over...", a positive example is (fox, jumps).
2. Negative Sampling: We also generate "negative" examples of words that don't appear in the context, like (fox, airplane).
3. The Trick: We train a simple neural network to distinguish between positive and negative pairs. The model's goal is to adjust its internal weights so that the dot product of vectors for positive pairs is high, and the dot product for negative pairs is low.
4. The Result: We don't actually care about the prediction task itself. We throw away the classifier part of the network and keep the learned weights. These weights, which have been optimized to understand which words appear together, form our embedding matrix—a lookup table where each row is the dense vector for a word.

This process transforms the unstructured relationships in text into the geometric relationships in the vector space we saw earlier.

Conclusion

In this lesson, we have opened the black box of LLMs to examine the first fundamental component: word representation. We've seen how simple methods like one-hot encoding are insufficient and how dense vector embeddings solve these problems by capturing rich semantic meaning in an efficient format.

Key Takeaways:

• Word embeddings are dense, lower-dimensional vector representations of words.
• They are based on the distributional hypothesis: words appearing in similar contexts have similar meanings and thus similar vectors.
• Embeddings capture complex semantic and syntactic relationships, which can be revealed through vector arithmetic (e.g., king - man + woman ≈ queen).
• They are learned via self-supervised training, often by setting up a "fake" prediction task (like in Word2Vec) where the learned model weights become the embeddings. This process effectively distills statistical patterns from a vast text corpus into a meaningful geometric space.

Next Lesson Preview

Word embeddings give us a powerful way to represent individual words. However, a sentence's meaning depends critically on the order and relationships between its words. For example, "man bites dog" is very different from "dog bites man," even though they use the same word embeddings.

In our next lesson, we will explore the self-attention mechanism. This is the revolutionary concept at the heart of the Transformer architecture that allows a model to weigh the importance of different words in a sequence when producing a representation, forming the basis for contextual understanding.