Text Chunk Vector Embeddings

Welcome back! In our previous lesson, we implemented semantic chunking, a technique that relies on the meaning of text to create coherent chunks. Central to that process was the concept of an "embedding," which we treated as a given tool to measure semantic similarity.

Today, we will open that black box. Our objective is to generate vector embeddings for text chunks using a modern embedding model. This is a fundamental step in any RAG system. As an AI Engineer, you won't just generate embeddings; you'll be responsible for selecting the right model based on critical trade-offs between cost, performance, and quality. We will explore these trade-offs and then implement two common approaches for generating embeddings: using a commercial API and running an open-source model locally.

Let's look at where this fits in our overall RAG architecture.

What is a Vector Embedding?

At its core, an embedding is a numerical representation of a piece of text (or other data like an image). An embedding model is a neural network trained to map semantically similar pieces of text to points that are close to each other in a high-dimensional vector space.

For a clear and concise explanation of this core concept, the following video from Hugging Face is an excellent starting point.

The key takeaway is that embeddings turn unstructured text into structured numerical data that we can perform mathematical operations on, like calculating similarity. This is the foundation of semantic search.

The Engineer's Dilemma: Choosing an Embedding Model

As a software engineer, you know that choosing a library or service involves weighing trade-offs. Selecting an embedding model is one of the most critical architectural decisions in a RAG system because, unlike a generative LLM, you can't easily swap it out. The vectors generated by one model are incompatible with those from another. Changing models requires re-embedding your entire document corpus, which can be costly and time-consuming.

The article "Embeddings Comparison" from MyEngineeringPath provides an excellent framework for making this decision.

The main decision boils down to Cloud APIs vs. Self-Hosted Open-Source models:

• Cloud APIs (e.g., OpenAI, Cohere, Voyage AI):
  • Pros: Easy to use (just an API call), no infrastructure to manage, highly available, and often offer state-of-the-art performance.
  • Cons: Pay-per-token pricing can get expensive at scale, data must be sent to a third-party service, and you are subject to vendor rate limits and potential model changes.
• Self-Hosted Open-Source (e.g., BGE, Nomic, GTE):
  • Pros: Full control over your data (sovereignty), no per-token cost (only fixed compute cost), and predictable model behavior (you pin the version).
  • Cons: You are responsible for setting up and maintaining the GPU infrastructure, scaling, and ensuring uptime.

For prototyping and most initial production systems, starting with a cost-effective API like text-embedding-3-small from OpenAI is a very common and pragmatic choice.

Generating Embeddings: The Implementation

Now, let's write some code. We'll implement both approaches: using the OpenAI API and using a local open-source model with the sentence-transformers library. Your basic Python knowledge will be sufficient here, as the libraries abstract away most of the complexity.

The following resource provides clear, commented code for exactly these two scenarios.

Let's summarize the key code patterns you've just reviewed.

Approach 1: OpenAI API

This is the simplest way to get started. After installing openai, the core logic is:

from openai import OpenAI




# The client uses the OPENAI_API_KEY environment variable by default
client = OpenAI()

def get_openai_embeddings(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    """Generates embeddings for a list of texts using OpenAI's API."""
    response = client.embeddings.create(
        model=model,
        input=texts
    )



    # The API returns data that might be out of order, so we sort by index
    # to ensure the order of embeddings matches the input texts.
    sorted_data = sorted(response.data, key=lambda x: x.index)
    return [item.embedding for item in sorted_data]




# Example usage with the chunks we created in the last lesson
text_chunks = [
    "The first rule of RAG is to have good chunks.",
    "The second rule of RAG is to use a good embedding model."
]

embeddings = get_openai_embeddings(text_chunks)

print(f"Generated {len(embeddings)} embeddings.")
print(f"Dimension of the first embedding: {len(embeddings[0])}")

Approach 2: Local Open-Source Model

This approach gives you more control and avoids API costs. Using the sentence-transformers library is the standard for this.

# You'll need to install with: pip install sentence-transformers torch
from sentence_transformers import SentenceTransformer




# The model is downloaded and cached automatically on first use
model = SentenceTransformer("all-MiniLM-L6-v2")

def get_local_embeddings(texts: list[str]) -> list[list[float]]:
    """Generates embeddings for a list of texts using a local model."""



    # The `encode` method handles batching internally for efficiency.
    # We set normalize_embeddings=True, which is standard practice for cosine similarity.
    embeddings = model.encode(texts, normalize_embeddings=True)
    return embeddings.tolist()




# Example usage
text_chunks = [
    "The first rule of RAG is to have good chunks.",
    "The second rule of RAG is to use a good embedding model."
]

embeddings = get_local_embeddings(text_chunks)

print(f"Generated {len(embeddings)} embeddings.")
print(f"Dimension of the first embedding: {len(embeddings[0])}")

Notice that while the underlying mechanics are different, the programming interface is conceptually similar: you pass a list of strings and get back a list of vectors (lists of floats).

Production Considerations

When moving from a script to a production system, a few more engineering details become important.

• Batching: Sending texts one by one to an embedding model is highly inefficient due to network latency (for APIs) and model loading overhead (for local models). Always process texts in batches. Both openai.embeddings.create and model.encode are designed to accept a list of texts. The optimal batch size depends on the model's context window and API limits (e.g., OpenAI allows up to 2048 texts per call).
• Dimensionality: Higher dimensions generally mean better quality but also larger storage costs and slower search times in your vector database. Models like OpenAI's text-embedding-3 series support "Matryoshka" embeddings, allowing you to request a smaller dimension (e.g., 1024 instead of 3072) without re-embedding, offering a great way to balance cost and quality.
• Asynchronous Processing: Embedding a large corpus of documents can take hours. This is a perfect use case for an asynchronous architecture, which we will cover later in the course. New documents can be added to a message queue, and a pool of workers can process them in batches to generate and store embeddings without blocking the main application.

Conclusion

In this lesson, we demystified vector embeddings and learned how to generate them. This is the crucial step that translates our text chunks into a format that a computer can search based on meaning.

Key Takeaways:

• Embeddings are numerical vector representations of text that capture semantic meaning.
• Choosing an embedding model is a critical architectural decision, balancing cost, quality, and operational control (API vs. self-hosted).
• For most projects, starting with a cost-effective API like text-embedding-3-small is a pragmatic choice. As your scale or data privacy needs grow, migrating to a self-hosted open-source model becomes viable.
• The implementation is straightforward using libraries like openai for APIs or sentence-transformers for local models.
• Batching is essential for efficient and scalable embedding generation in a production environment.

We have now learned how to chunk our documents semantically and how to convert those chunks into vector embeddings. The final step in preparing our knowledge base is to store these vectors in a way that allows for fast and efficient similarity search. In our next lesson, we will do just that by learning how to index vector embeddings into a vector database.