Text Embeddings with Transformers

1. The Challenge of Context: From Static to Dynamic Embeddings

At its core, an embedding is a numerical representation—a vector of numbers—that captures the semantic meaning of a piece of text. Early models like Word2Vec were a major breakthrough because they learned vector representations where words with similar meanings were located close to each other in the vector space.

However, they had a critical flaw: they were static. A word had exactly one vector representation, regardless of how it was used.

Consider these two sentences:

• "He deposited his money in the bank."
• "The canoe drifted to the river bank."

A static model would assign the exact same vector to "bank" in both cases, completely missing the contextual difference between a financial institution and a river's edge. This ambiguity is a significant problem for any system that needs a nuanced understanding of language.

To see why this is a problem and to get a high-level overview of how transformers solve it, please watch the following segment from the video "Transformer Embeddings - EXPLAINED!".

As the video explains, transformer models were designed to overcome this very problem by generating contextualized embeddings. The vector for "bank" is no longer fixed; it is dynamically computed based on the other words in the sentence. Let's dive into how it does that.

2. The Engine of Context: Self-Attention

The magic behind contextualization in transformers is a mechanism called self-attention. The core idea is surprisingly intuitive: to understand a word's meaning in a sentence, the model should "pay attention" to the other words in that same sentence.

For example, when processing the word "bank" in "He deposited his money in the bank," the self-attention mechanism learns to place high importance on "money" and "deposited," nudging the final vector for "bank" towards a financial meaning.

The process of creating these contextualized embeddings inside a transformer encoder block (the building block of models like BERT) can be broken down into a few key steps.

Step 1: Input Representation

Before the self-attention mechanism can work its magic, the model needs to know two things about each word: its general meaning and its position in the sentence.

• Token Embeddings: The input text is first tokenized (split into words or sub-words), and each token is mapped to a static embedding, similar to a Word2Vec vector.
• Positional Encodings: Since transformers process all words in a sentence simultaneously (unlike RNNs), they have no inherent sense of word order. To solve this, a "positional encoding" vector is added to each token's embedding. This vector provides information about the word's position, ensuring the model knows that "man bites dog" is different from "dog bites man."

Step 2: The Self-Attention Calculation

This is where the context is incorporated. For each word in the sentence, the model calculates a new, context-aware vector. This is done by taking a weighted average of all the other word vectors in the sentence. The "weights" determine how much attention each word should pay to every other word.

To understand this process from first principles, let's read through the following article which frames the invention of self-attention as a logical solution to the context problem.

As the article explained, the attention weights are not random. They are calculated using three special vectors derived from each word's initial embedding:

• Query (Q): Represents the current word that is "looking for" context. Think of it as this word asking, "Who in this sentence is relevant to me?"
• Key (K): Represents a word that can provide context. It acts like a label that the Query vector can compare itself against.
• Value (V): Represents the actual content or meaning of a word. This is the vector that will be contributed to the final output if the word is deemed relevant.

The process for a single word is as follows:

1. Calculate Similarity Scores: The Query vector of the word in focus is compared with the Key vector of every other word in the sentence (including itself). This comparison is typically a dot product. A high score means high relevance.
2. Calculate Attention Weights: The similarity scores are passed through a softmax function, which converts them into a set of positive weights that sum to 1. These are the attention weights.
3. Compute the Final Vector: The final, contextualized vector for the word is a weighted sum of all the Value vectors in the sentence, using the attention weights calculated in the previous step.

The following image provides a detailed diagram of this entire flow.

Step 3: Refinements for Richer Context

The actual mechanism in models like BERT is slightly more complex to improve its power:

• Multi-Head Attention: Instead of doing this calculation once, the model does it multiple times in parallel, each with different, independently learned Q, K, and V transformations. Each "head" can learn to focus on different types of relationships (e.g., one head might focus on syntactic relationships, another on semantic ones). The results are then combined to produce an even richer representation.
• Stacked Layers: A transformer encoder consists of multiple identical layers stacked on top of each other. The output of one layer becomes the input to the next. This allows the model to build progressively more abstract and complex representations of the text.

3. From Word Tokens to Sentence Chunks

The self-attention process we've described gives us a highly contextualized vector for every single token in our input text. However, for our RAG system's retrieval phase, we need to compare a user's query against chunks of our documents. This requires a single vector that represents the meaning of the entire chunk (e.g., a whole sentence or paragraph).

This is achieved by adding an aggregation or pooling step after the transformer encoder. A common strategy is mean pooling, where you simply take the average of all the output token embeddings to create a single "sentence embedding".

Models like Sentence-Transformers are specifically fine-tuned for this purpose. They take a pre-trained transformer like BERT and train it further on tasks that require understanding sentence-level meaning, ensuring that the final pooled output is a high-quality sentence embedding.

The following video explains this crucial final step.

This final sentence embedding is the dense vector that we will store in our vector database. When a user asks a question, we will perform the same process on their query to generate a query vector, and then search the database for the most similar document chunk vectors.

Conclusion

You now have a solid conceptual framework for how modern NLP models convert raw text into powerful, context-aware dense vectors.

Here are the key takeaways:

• Context is Key: Static embeddings (like Word2Vec) are limited because they can't distinguish between different meanings of the same word. Transformer-based models solve this by creating dynamic, contextualized embeddings.
• Self-Attention is the Engine: The core mechanism is self-attention, where each word's representation is formed by a weighted average of all other words in the sequence. The weights are determined by relevance, calculated using Query, Key, and Value vectors.
• From Tokens to Sentences: A base transformer outputs an embedding for each token. To get a single embedding for a sentence or chunk, an aggregation step like pooling is applied to the token embeddings.

In our next lesson, we will build on this foundation. We'll move from theory to practice by comparing popular, pre-trained embedding models (like those from Sentence-Transformers and OpenAI) and analyze their trade-offs in terms of performance, cost, and suitability for our RAG project.