Word Embeddings: A Superior Representation

From Words to Numbers

Machine learning models operate on numbers, not text. Therefore, the first challenge in any NLP task is to convert words into a numerical format. A simple idea might be to assign a unique integer to each word in our vocabulary—for instance, cat -> 1, dog -> 2, and so on.

However, this method, known as integer encoding, introduces an unintended and problematic side effect: it imposes an artificial order. A model might interpret that dog (2) is "greater than" cat (1) or that the "distance" between cat (1) and dog (2) is the same as between dog (2) and horse (3). These numerical relationships are arbitrary and don't reflect any real-world meaning, which can confuse a model and hinder its ability to learn.

The following video provides a clear explanation of this problem.

One-Hot Encoding: A Step Up, But Flawed

To solve the artificial ordering problem, data scientists developed one-hot encoding. The idea is to create a vector for each word that is as long as the entire vocabulary. This vector is composed entirely of zeros, except for a single '1' at the index that uniquely corresponds to that word.

For a tiny vocabulary of {king, queen, prince, princess}:

• king: [1, 0, 0, 0]
• queen: [0, 1, 0, 0]
• prince: [0, 0, 1, 0]
• princess: [0, 0, 0, 1]

This approach successfully removes the artificial ordering. Now, no word is numerically "greater" than another. However, one-hot encoding introduces its own set of significant disadvantages, which ultimately make it impractical for the large vocabularies used in modern NLP.

Let's examine these drawbacks in detail.

As the article points out, the main problems with one-hot encoding are:

1. High Dimensionality and Sparsity: A typical English vocabulary for a large model might contain 50,000 to 100,000 words. This means each word would be represented by a 50,000-dimensional vector that is almost entirely filled with zeros. As a data scientist, you know that high-dimensional, sparse data is computationally expensive to process and requires a massive amount of memory. Imagine loading a lookup table with millions of these huge vectors into memory for your model.

2. Lack of Semantic Relationships: This is the most critical failure of one-hot encoding. Because the vector for each word has a '1' in a unique dimension, all one-hot vectors are orthogonal to each other. Mathematically, the dot product of any two different word vectors is zero (e.g., king · queen = 0). This means the representation contains no information about the relationships between words. The model has no inherent clue that king is more similar to queen than it is to apple. It must learn every single word relationship from scratch, which is incredibly inefficient and requires vast amounts of data.

The following video segment reinforces these points visually and conceptually.

The Solution: Dense Embeddings

To overcome these challenges, we use word embeddings, which are dense, low-dimensional vectors.

• Dense: Unlike the sparse one-hot vectors, most of the values in an embedding vector are non-zero.
• Low-dimensional: Instead of 50,000+ dimensions, embeddings typically have between 50 and 1000 dimensions—a dramatic reduction.

The image below provides a great visual summary of the contrast.

Crucially, these dense vectors are not just random numbers. They are learned from text data in such a way that they capture the semantic relationships between words. Words with similar meanings or that are used in similar contexts will have vectors that are "close" to each other in the multi-dimensional vector space.

This property is what makes embeddings so powerful. The geometry of the embedding space mirrors the semantic relationships of the language. This allows a model to generalize. For example, if a model learns something about the word "excellent" from the training data, it can apply some of that learning to the word "superb" because their vectors will be close together in the embedding space.

The most famous illustration of this is the concept of word analogies through vector arithmetic. In a well-trained embedding space, the vector operation vector('king') - vector('man') + vector('woman') results in a vector that is very close to vector('queen').

This shows that the model has learned abstract relationships, like gender, from the data and encoded them into the directions of the vectors.

The final video clip for this lesson provides a fantastic intuition for these concepts.

Conclusion

In this introductory lesson, we've established the fundamental "why" behind modern word representations. Let's summarize the key advantages of dense embeddings over one-hot encoding:

• Efficiency: Dense embeddings are much lower-dimensional (e.g., 300 vs. 50,000+), making them significantly more efficient in terms of both computation and memory.
• Semantic Richness: Unlike orthogonal one-hot vectors, embeddings capture rich semantic relationships. Proximity in the vector space corresponds to similarity in meaning, allowing models to understand concepts like synonyms and analogies.
• Generalization: Because embeddings capture semantic relationships, models can generalize what they've learned about one word to other, similar words. This leads to more robust performance and better learning from less data compared to what would be needed with one-hot vectors.

You now have a solid conceptual foundation for why dense vectors are the standard for representing text in machine learning. You've moved from simply using embeddings to understanding the critical problems they solve.

In our next lesson, we will build upon this foundation by exploring how these powerful dense vectors are created. We'll dive into the architecture of one of the most influential early models for learning embeddings: Word2Vec.