Welcome to the second module of our course! In the first module, you mastered the fundamentals of programmatic LLM interaction, learning how to send requests, control generation, and, most importantly, parse the model's output into reliable, structured data. You now have the skills to make an LLM a predictable component in a larger software system.
Now, we shift our focus from the "Generation" part of RAG to the "Retrieval" part. The central challenge in retrieval is finding information that is semantically relevant to a user's query, not just information that shares the same keywords. This lesson tackles the foundational concept that makes this possible: text embeddings. We will explore how we can distill the rich, nuanced meaning of language into a format that computers excel at processing: numerical vectors. Understanding this translation from words to vectors is the first and most crucial step toward building the powerful search systems that underpin RAG.
The Problem: How Do Computers "Understand" Text?
As a software engineer, you're used to data having a clear, structured representation—integers, booleans, objects with defined properties. But what about text? How do we represent raw, unstructured sentences in a way that captures their meaning?
A simple first thought might be to assign a unique ID number to each word. However, this approach quickly breaks down because the numerical values themselves would imply relationships that don't exist in the language. For instance, if "good" is token #6, "great" is #21, and "bad" is #22, a model might incorrectly infer that "great" and "bad" are more similar than "good" and "great".
To see a great breakdown of why these simple approaches are insufficient, let's watch a segment from the Under The Hood channel.
This video does an excellent job of walking through the limitations of early text representation methods.
Please watch the section from the beginning up to the discussion of engrams. Pay close attention to the critique of three methods: simple token numbers, one-hot encoding, and bag-of-words. Notice how they all fail to capture semantic relationships between words.
As the video explains, techniques like one-hot encoding and bag-of-words either create enormous, sparse vectors or lose word order and context. They treat words as discrete, independent items, failing to see that "laptop" and "computer" are related, while "laptop" and "pizza" are not. This is often called the semantic gap: the disconnect between the words on the page and the meaning they convey. To build an effective retrieval system, we need to bridge this gap.
The Solution: Dense Embeddings
The solution is to represent text not with sparse vectors tied to specific words, but with dense vectors where each number in the vector represents some latent, abstract feature of the text's meaning. These vectors are known as text embeddings.
Unlike a one-hot vector which might have 50,000 dimensions (one for each word in a vocabulary) with only a single '1', a dense embedding might have a few hundred to a few thousand dimensions, and nearly all of them will have non-zero values.
How Text Embeddings Actually Work Under the Hood | Let's Data Science
This article provides a very clear, concise answer to a common interview question that perfectly summarizes the core difference between old and new methods.
Please read the answer to the first interview question, "What is the difference...". This captures the essence of why dense embeddings are a fundamental leap forward.
An embedding model, typically a deep neural network, is trained on a massive corpus of text (like the entire internet). Through this training, it learns to convert any piece of text—a word, a sentence, or a whole document—into a fixed-size vector of floating-point numbers. The key property of these vectors is that semantic similarity corresponds to geometric proximity. Texts with similar meanings will have vectors that are close to each other in this high-dimensional space.
The "Semantic Space"
This high-dimensional space where the embedding vectors "live" is called a semantic space. The dimensions of this space aren't explicitly defined by humans; they are abstract features learned by the model. One dimension might loosely correspond to "formality," another to "object vs. action," and another to "animal-relatedness," but in a way that is far more complex and nuanced than we can label.
What are vector embeddings? A complete guide [2025]
The Meilisearch blog provides a fantastic, non-technical explanation of the semantic space concept.
First, read the introductory sections "What are vector embeddings?" and "How do they work?" for a solid definition. Then, focus on the section "What is the semantic space?", including the illustrative example with the "feline," "juvenile," and "canine" axes. This provides a brilliant intuition for how this space is structured.
As the article explains, the simple example with axes for canine or feline is a helpful mental model. In reality, the vectors are composed of floating-point numbers, capturing subtle shades of meaning across hundreds or thousands of these learned, latent dimensions.
Meaning as Geometry: Vector Arithmetic
The most remarkable property of this semantic space is that the relationships between concepts are captured by the geometry of their vectors. This allows us to perform a kind of "conceptual math." The classic example is the analogy king is to queen as man is to woman. In the embedding space, this relationship holds true for the vectors as well.
If you take the vector for king, subtract the vector for man, and add the vector for woman, the resulting vector will be extremely close to the vector for queen.
vector('king') - vector('man') + vector('woman') ≈ vector('queen')
This isn't a cherry-picked oddity; it's a fundamental property that demonstrates the model has learned rich semantic relationships.
The same "Under The Hood" video provides a wonderful visual explanation of this vector arithmetic.
Watch the segment from word embeddings in detail. It starts with a simple 2D example and then demonstrates the king - man + woman = queen and Beijing - China + Japan = Tokyo analogies with concrete vector operations.
This geometric structure is what allows an embedding-based search system to understand analogies and relationships that are completely invisible to keyword matching.
From Static Words to Dynamic Context
The idea of embeddings is not new. Early models like Word2Vec and GloVe (from around 2013-2014) were revolutionary because they were the first to produce these dense vectors and demonstrate the vector arithmetic properties. However, they had a significant limitation: they produced a single, static vector for each word. The word "bank" would have the exact same vector in "river bank" and "investment bank."
Modern embedding models, based on the Transformer architecture (the same technology behind models like GPT), are contextual. They generate a different embedding for a word or sentence based on the surrounding text. The vector for "update" in "apply the software update" will be different from the vector for "update" in "give me an update on your progress." For RAG applications, where the precise meaning of a document chunk or a user query is critical, these contextual embeddings are essential.
How Text Embeddings Actually Work Under the Hood | Let's Data Science
This section of the "Let's Data Science" article provides a concise history of the evolution from static to contextual embeddings.
Please read the section From Word2Vec to sentence transformers. You don't need to memorize the model names, but focus on the key conceptual shift from static embeddings (Word2Vec, GloVe) to contextual embeddings (ELMo, BERT, and modern models).
Conclusion
In this lesson, you've moved from the world of text as simple strings to the world of text as rich, meaningful vectors. This is the conceptual bedrock upon which all modern semantic search and RAG systems are built.
Here are the key takeaways:
- Embeddings as a Solution: Text embeddings are dense numerical vectors that capture the semantic meaning of text, bridging the "semantic gap" left by traditional keyword-based methods.
- The Semantic Space: These vectors exist in a high-dimensional space where distance represents similarity. Texts with similar meanings are located close to each other.
- Structured Meaning: This space isn't random; it has a geometric structure that reflects real-world relationships, enabling "vector arithmetic" like
king - man + woman ≈ queen. - The Importance of Context: Modern embedding models are contextual, producing different vectors for the same text depending on its surrounding words, which provides a much more nuanced representation of meaning.
You now understand what embeddings are and why they are so powerful. In the next lesson, we will get practical. You will learn how to take any piece of text and generate your own embeddings using a pre-trained model through an API, taking your first hands-on step towards building a retrieval system.