Choosing the Right Embeddings & Chunking: A Cost-Benefit Approach

A Framework for Decision-Making

To make our initial choices, we'll follow a structured framework that balances three key factors for both the embedding model and the chunking strategy:

1. Performance: How well does it accomplish the task (retrieval quality)?
2. Cost: What are the financial and computational expenses?
3. Complexity: What is the engineering overhead for implementation and maintenance?

Let's start with selecting our embedding model.

Part 1: Selecting an Embedding Model

Choosing an embedding model is one of the most impactful decisions for a RAG system. It directly influences retrieval accuracy, cost, and infrastructure requirements.

To begin, let's get a high-level overview of the factors involved.

The video outlines the main decision points. Now, let's apply them to our project using a more detailed cost-benefit lens.

Analysis and Recommendation

Drawing from the resources, let's analyze our options for the project.

1. Performance:
Public benchmarks give us a great starting point. The Massive Text Embedding Benchmark (MTEB) from Hugging Face is the industry standard for comparing open-source models.

These charts show a clear trend: you can get excellent performance from both high-cost proprietary models and free open-source models. The key is finding the sweet spot. For our project, which involves processing dense academic texts, retrieval performance is paramount.

2. Cost & Complexity:

• Proprietary API Models (e.g., OpenAI text-embedding-3-large)
  • Pros: State-of-the-art performance, zero setup/maintenance overhead.
  • Cons: High and unpredictable costs (pay-per-token), vendor lock-in, rate limits. Every time we re-index our documents during development, we incur a cost.
• Self-Hosted Open-Source Models (e.g., BAAI bge-large-en-v1.5, all-MiniLM-L6-v2)
  • Pros: No per-token costs, full control, encourages learning the MLOps side of deployment.
  • Cons: Requires local compute resources (CPU/GPU) and engineering effort to manage.

Initial Decision:

For the initial development phase of our project, an open-source model is the most strategic choice.

• Cost-effectiveness: It allows for unlimited experimentation and re-indexing without incurring API fees.
• Alignment with Goals: It forces us to engage with the operational aspects of running a model locally, directly supporting your goal of moving beyond "notebook-level" scripting.

A great starting candidate is BAAI/bge-small-en-v1.5.

• Performance: It's a strong performer on the MTEB leaderboard for its size.
• Size: With a dimension of 384, it's memory-efficient for vector storage and fast for similarity search.
• Complexity: It's small enough to run quickly on a standard CPU, removing any dependency on a GPU for the initial build-out.

We will keep a high-end model like OpenAI text-embedding-3-large in mind as a benchmark. Once our system is more mature, we can run an evaluation to see if its performance uplift justifies the cost.

Part 2: Formulating an Initial Chunking Theory

In the previous lesson, you explored a hierarchy of chunking strategies. Now, we must select a starting "theory" that is best suited for our specific documents—PDFs of chemical engineering papers.

Let's consult a practical decision framework.

Analysis and Recommendation

Our content (academic papers) has specific characteristics:

• Structure: Clear hierarchy (Abstract, Introduction, Methods, Results, Discussion), section headers.
• Modality: Mixed content, including prose, tables (e.g., experimental data), figures (e.g., process flow diagrams), and equations.
• Queries: Likely to be a mix of factoid ("What catalyst was used?") and analytical ("Summarize the findings regarding reaction kinetics.").

Based on this and the research findings, we can formulate our initial chunking theory.

1. The Default Choice: Recursive Character Splitting
   As both the Firecrawl article and Chroma's research (which you saw in the video The BEST Way to Chunk Text for RAG) conclude, RecursiveCharacterTextSplitter is the best place to start. It's computationally cheap and surprisingly effective.

   • Our Plan: We'll use it as our baseline for all prose content.
   • Initial Parameters: Based on the research, we'll start with a chunk size of ~512 tokens and an overlap of ~10% (50 tokens). This is a balanced choice for handling both factoid and analytical queries.

2. The High-Potential Contender: Page-Level Chunking
   NVIDIA's research, cited in the Firecrawl article, found that page-level chunking performed best for structured PDFs. This makes intuitive sense for academic papers, where authors often organize figures, tables, and related text on a single page.

   • Our Plan: While more complex to implement initially, we will earmark page-level chunking as the first alternative strategy to benchmark against our recursive baseline.

3. The Acknowledged Limitation: Multimodal Content
   Our simple chunking theory for text doesn't address tables and figures. A production-grade system would need a more sophisticated, structure-aware approach (Level 3 from our previous lesson) to handle these. For instance, using a tool like unstructured.io to extract tables as HTML/Markdown or generating image summaries.

   • Our Plan: For our initial implementation, we will focus on text extraction. However, our system design must acknowledge this limitation, and we will plan to incorporate multimodal parsing in a future iteration.

Initial Chunking Theory:

We will begin with a Recursive Character Splitting strategy with a 512-token chunk size and 50-token overlap as our primary method. This provides a robust and efficient baseline. We will design our evaluation pipeline with the explicit goal of A/B testing this against a Page-Level Chunking strategy to see if leveraging the PDF's visual layout provides a performance lift for our specific documents. Handling of tables and figures will be deferred but noted as a key area for future improvement.

Conclusion: Our Starting Blueprint

Congratulations on completing the foundational module! You've moved from understanding individual components to formulating a complete, justifiable plan for the first phase of our RAG project.

Key Takeaways (Your Initial Design):

• Embedding Model: We will start with an open-source model like BAAI/bge-small-en-v1.5. It offers a strong balance of performance, zero financial cost for development, and an opportunity to build MLOps skills.
• Chunking Strategy: We will start with Recursive Character Splitting (~512 token chunk size, ~10% overlap) as a robust baseline, with a plan to evaluate Page-Level Chunking as a promising alternative for our PDF documents.
• Engineering Principle: This is our starting point. Every choice is a hypothesis to be tested. This mindset is what separates prototyping from building reliable, production-ready systems.

Preview of the Next Lesson:

In our next module, "Advanced Ingestion and Retrieval Preparation," we will roll up our sleeves and start coding. You will implement the first part of this blueprint: creating a configurable ingestion pipeline that uses the recursive chunking strategy to process documents and prepares them for embedding.