Text Embeddings with Pre-trained Models via API

Generating Embeddings with the OpenAI API

The most direct way to start generating high-quality embeddings is by using a specialized, pre-trained model through an API. OpenAI provides industry-standard models that are easy to access and highly performant. The overall process is simple: you send a string of text to an API endpoint, and the model returns its vector representation.

To accomplish this in Python, we'll use the openai library. The interaction follows a standard client-server pattern you're likely familiar with.

The Core API Call

Let's look at the specifics of the API call. You'll need to instantiate a client, then call the embeddings.create method, providing the text you want to embed and specifying which model to use.

As you saw in the documentation, the primary models you'll encounter are from the text-embedding-3 family. For many RAG applications, text-embedding-3-small offers a fantastic balance of high performance and low cost. The API response is an object that contains the embedding vector itself within response.data[0].embedding.

A Practical Walkthrough

Let's consolidate this into a reusable Python function and see it in action. The following video provides a clear, step-by-step demonstration of setting up the client, creating a function to generate an embedding, and then applying that function to a real dataset.

Here is a summary of the coding pattern, updated to use the latest text-embedding-3-small model. Note that you must have your OpenAI API key set as an environment variable (OPENAI_API_KEY) for this code to run.

# First, ensure you have the library installed:
# pip install openai pandas

import os
import openai
import pandas as pd




# The client will automatically pick up the OPENAI_API_KEY environment variable.
client = openai.OpenAI()

def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]:
    """
    Generates an embedding for a given text using a specified OpenAI model.
    """



    # Defensive programming: ensure text is not empty and is a string
    if not text or not isinstance(text, str):
        return None
        
    text = text.replace("\n", " ") # Models are sensitive to newlines
    
    try:
        response = client.embeddings.create(input=[text], model=model)
        return response.data[0].embedding
    except Exception as e:
        print(f"An error occurred: {e}")
        return None




# Example 1: Embedding a single sentence
sentence = "The quick brown fox jumps over the lazy dog."
embedding_vector = get_embedding(sentence)
if embedding_vector:
    print(f"Embedding vector for the sentence (first 5 dimensions): {embedding_vector[:5]}")
    print(f"Vector dimensionality: {len(embedding_vector)}")




# Example 2: Applying the function to a pandas DataFrame
data = {'text': ["What is the capital of France?", "A delicious recipe for pasta carbonara.", "The principles of quantum mechanics."]}
df = pd.DataFrame(data)

df['embedding'] = df['text'].apply(get_embedding)

print("\nDataFrame with embeddings:")
print(df)

This pattern is a cornerstone of preparing data for a RAG system. You now have a robust way to convert any collection of text documents into a list of corresponding embedding vectors.

Beyond OpenAI: Using Open-Source Models with Hugging Face

While OpenAI provides excellent models, the field of AI is vast and includes a thriving open-source community. Hugging Face is the leading platform for this community, hosting tens of thousands of models. Their Inference API allows you to use many of these models, including powerful embedding models, through a unified interface.

Exploring this ecosystem is key to gaining a deeper understanding and avoiding vendor lock-in. The core idea is the same—an API call that returns a vector—but you have a much wider selection of models to choose from.

This video from Hugging Face provides a perfect introduction to their Inference Providers and demonstrates how to generate embeddings with an open-source model.

As you saw, the workflow is remarkably similar to using the OpenAI API. You install a library (huggingface_hub), get an API key, instantiate a client, and call a method. Here's the key code snippet from the video for easy reference:

# First, ensure you have the library installed:
# pip install huggingface_hub

from huggingface_hub import InferenceClient




# You'll need to set your HF_TOKEN environment variable
client = InferenceClient()




# Example using a popular open-source model
# You can find model IDs on the Hugging Face Hub
model_id = "sentence-transformers/all-MiniLM-L6-v2"
text = "Today is a sunny day and I will get some ice cream."

embedding_vector_hf = client.feature_extraction(model=model_id, text=text)




# The structure of the response might differ slightly by model/task.
# For feature_extraction, the result is often the list of floats directly.
print(f"Hugging Face embedding (first 5 dimensions): {embedding_vector_hf[0][:5]}")
print(f"Vector dimensionality: {len(embedding_vector_hf[0])}")

The main differences are the method name (feature_extraction instead of embeddings.create) and the way you specify the model (using its public Hugging Face ID). The underlying engineering pattern, however, is identical. This flexibility allows you to experiment with different models that may be better suited or more cost-effective for specific tasks.

Practical Considerations: Cost and Dimensionality

When generating embeddings at scale, two factors become very important: cost and performance.

1. Cost and Tokenization: APIs charge based on the number of tokens in your input text. A token is a common sequence of characters; for English, one token is roughly 4 characters or 0.75 words. Before sending large batches of text to be embedded, it's wise to estimate the token count to manage costs. The OpenAI tiktoken library is the standard tool for this.

2. Embedding Dimensionality: An embedding's "size" or "dimensionality" is the number of floating-point numbers in its vector. For text-embedding-3-small, this is 1536. Larger dimensions can capture more nuance but increase storage costs, memory usage, and the computational cost of similarity searches. OpenAI's newer models allow you to reduce the dimensionality via the dimensions parameter in the API call, offering a direct trade-off between performance and cost.

For a deeper look at these practical details, the OpenAI documentation provides excellent guidance.

Conclusion

In this lesson, you have taken the crucial step from concept to code. You can now programmatically convert text into meaningful numerical vectors, the essential raw material for any RAG system.

Here are your key takeaways:

• API-based Generation: Embeddings are generated by sending text to pre-trained models via API calls.
• OpenAI Workflow: Using the openai library, you can call models like text-embedding-3-small to get high-quality embeddings. The core pattern is client.embeddings.create().
• Open-Source with Hugging Face: The huggingface_hub library and Inference API provide a unified way to access a vast ecosystem of open-source embedding models using the client.feature_extraction() method.
• Practical Trade-offs: Real-world application requires balancing model choice, cost (managed by monitoring token counts), and performance (which can be tuned via dimensionality).

You've successfully turned text into vectors. But what makes these vectors so useful is our ability to compare them. In our next lesson, we will explore the mathematical operations, like cosine similarity and dot product, that allow us to quantify the "distance" between these vectors and, by extension, the semantic similarity between the original texts. This is the mechanism that powers semantic search.