Welcome to our fifth lesson in the foundational module on LLMs. In our previous session, we delved into the parameters like temperature, top-p, and top-k, which give you fine-grained control over the style and creativity of a model's generated text. We learned how to steer the model towards predictable, factual answers or towards more novel and surprising ones.
Now, we pivot from controlling how a model responds to influencing what it knows. A fundamental limitation of LLMs is that their knowledge is frozen at the time of their training. They don't know about events after their knowledge cut-off date, and more importantly, they don't know about your private, proprietary data—like your company's logistics procedures or real-time shipment statuses. This lesson introduces the core technology that allows us to bridge this knowledge gap.
Today, we will explore embeddings and vector similarity. You will learn what embeddings are, how these numerical representations capture the semantic meaning of data, and how the principle of vector similarity enables a powerful technique called semantic search. This concept is the cornerstone of Retrieval-Augmented Generation (RAG), a pattern we will master in a later module to build agents that can reason about your specific data.
What Are Embeddings? The Language of Meaning
At a high level, an embedding is a process of converting data—such as text, an image, or a sound—into a list of numbers, called a vector. These aren't just any numbers; they are coordinates that place the data in a high-dimensional "meaning space." The magic of embeddings, generated by specialized AI models, is that items with similar meanings are placed close together in this space.
Imagine you're trying to describe movies. You could use a few dimensions: "How funny is it?", "How scary is it?", "How romantic is it?". A comedy would have a high score on the "funny" axis, a horror film a high score on the "scary" axis, and a horror-comedy would fall somewhere in between. Real-world embedding models do this with hundreds or even thousands of dimensions, capturing incredibly nuanced aspects of meaning that are learned automatically from vast amounts of data.
This video provides an excellent and intuitive explanation of this core idea.
Embeddings & Vector Databases Explained
This video from LearnThatStack, "Embeddings & Vector Databases Explained," breaks down these concepts with clear analogies and visuals.
Please watch the first few minutes, from the start until 03:21. Pay close attention to: The idea of plotting words like "king" and "queen" in a 2D space to show their relationship. The movie analogy, which illustrates how multiple dimensions capture different aspects of meaning. The famous king - man + woman = queen equation, which demonstrates that these vectors capture true semantic relationships. The brief explanation of how these embeddings are learned by neural networks.
The key takeaway is that embeddings translate the fuzzy, human concept of "meaning" into a mathematical format that computers can work with. Similar concepts result in similar vectors.
Measuring Similarity: The Cosine Rule
How do we mathematically define "closeness" in this high-dimensional space? While several methods exist, the most common for text-based tasks is cosine similarity.
Instead of measuring the straight-line (Euclidean) distance between the points, cosine similarity measures the angle between the vectors.
- If two vectors point in the same direction, the angle between them is 0°, and their cosine similarity is 1 (identical).
- If they are orthogonal (unrelated), the angle is 90°, and their similarity is 0.
- If they point in opposite directions, the angle is 180°, and their similarity is -1 (opposite).
This is powerful because it focuses on the direction (the semantic content) and ignores the magnitude (which can be affected by things like sentence length).
The following image provides a more detailed, quantitative look at cosine similarity.
As a .NET developer, it can be helpful to see what this looks like in practice. The following article includes a simple C# implementation of the cosine similarity calculation, connecting the mathematical theory to code you can write.
Semantic Kernel Memory: Vector Stores, Embeddings, and Semantic Search - DEV Community
This article from the DEV Community provides a great overview of memory in Semantic Kernel, starting with the fundamentals of embeddings and similarity.
Please read the section Understanding Embeddings. Focus on the C# code block. You'll see how two semantically similar phrases ("The cat sat on the mat" and "A feline rested on the rug") produce a high similarity score, while unrelated phrases have a low score. The CosineSimilarity function implements the math we just discussed.
From Similarity to Semantic Search
Now we can connect these concepts to build a powerful search system. Semantic search is the process of searching for information based on conceptual meaning rather than exact keyword matches.
Here’s how it works:
- Indexing: You take your corpus of documents (e.g., your logistics policies, product descriptions, internal documentation) and generate an embedding vector for each piece of text. You store these vectors in a specialized database.
- Querying: A user submits a query (e.g., "How long do I have to send something back?").
- Embedding the Query: You use the exact same embedding model to convert the user's query into a vector.
- Similarity Search: You then calculate the cosine similarity between the query vector and every vector in your database.
- Retrieval: The document(s) with the highest similarity scores are the most semantically relevant results.
Notice that the query "How long do I have to send something back?" might not share any keywords (besides "to") with the best-matching document, which could be titled "30-Day Refund Policy". Semantic search understands the user's intent.
The LINK article we looked at earlier demonstrates this perfectly.
Semantic Kernel Memory: Vector Stores, Embeddings, and Semantic Search - DEV Community
Let's return to the same article to see semantic search in action.
Read the section titled Semantic Search. The code example searches for "How long do I have to return something?" and correctly retrieves the "refund-policy" document with a high relevance score, demonstrating the power of this approach.
The Need for Speed: Vector Databases
Calculating the similarity between a query vector and a few dozen other vectors is trivial. But what about millions or billions? A brute-force approach, comparing one-by-one, becomes computationally impossible.
This is where vector databases come in. These are databases purpose-built to store, index, and query vast numbers of embedding vectors at incredible speed. Instead of an exhaustive search, they use sophisticated Approximate Nearest Neighbor (ANN) algorithms to rapidly find vectors that are "close enough" to the query vector.
One of the most popular algorithms is HNSW (Hierarchical Navigable Small World). Think of it like a GPS. To get from your house to a specific address across the country, you don't check every local street. You first jump on highways (long-range connections) to get to the right city, then use major roads to get to the right neighborhood, and only then navigate the local streets (short-range connections) to find the exact address. Vector database indexes work on a similar principle, allowing them to narrow the search space dramatically.
For your work with GCP, the premier service for this is Vertex AI Vector Search. It's a managed, scalable service that handles the complexity of indexing and serving embeddings for you.
What is a vector database & how does it work? | Google Cloud
This Google Cloud article provides an excellent overview of vector databases and their applications.
Please read these two sections: How does a vector database work? This section explains the key mechanisms, including Nearest Neighbor Search (KNN), Approximate Nearest Neighbors (ANN), and indexing. Use cases for vector databases. Pay special attention to the first use case, Retrieval-augmented generation (RAG), as this is a central pattern for building the agents in this course.
Conclusion
In this lesson, we have demystified one of the most powerful concepts in modern AI. You now understand the end-to-end flow: from converting raw data into meaningful numerical representations to using vector similarity to find relevant information at scale.
Here are the key takeaways:
- Embeddings are vector representations of data (like text or images) that capture semantic meaning.
- Cosine Similarity is a common metric used to measure the closeness of two vectors by calculating the angle between them, indicating how similar their meaning is.
- Semantic Search leverages embeddings and similarity metrics to find information based on intent and meaning, not just keywords.
- Vector Databases, like Google's Vertex AI Vector Search, use Approximate Nearest Neighbor (ANN) algorithms to perform this similarity search efficiently across massive datasets.
- This entire process is the foundation for giving LLMs access to external, real-time, or proprietary knowledge, a pattern known as Retrieval-Augmented Generation (RAG).
In our next lesson, we will examine some of the inherent weaknesses of LLMs. We'll discuss common failure modes like hallucinations, prompt injection, and stale knowledge. Understanding these problems will further motivate why techniques like RAG, which you now have the foundational knowledge for, are so critical for building reliable, production-grade AI agents.