Text Chunking Strategies: Impact on Context and Retrieval

The Problem: The "Fixed-Size Trap" and the Chunking Trilemma

The most straightforward way to chunk a document is to split it into fixed-size pieces. While simple, this approach is often destructive because it has no regard for the content's structure. It can split sentences, break apart logical arguments, and separate a question from its answer.

This leads us to what is often called the Chunking Trilemma. When designing a chunking strategy, you must balance three competing goals:

1. Semantic Coherence: Chunks should contain complete, self-contained ideas.
2. Optimal Size (Granularity): Chunks must be large enough to provide context but small enough to be specific and fit within model limits.
3. Computational Efficiency: The time and cost required to process documents into chunks.

There is no single "best" strategy. The right choice depends on your document types, the kinds of questions users will ask, and your performance requirements. Let's explore the options.

Level 1 & 2: From Naive Splitting to a Smart Baseline

To understand the spectrum of chunking strategies, we will follow a "levels" framework, starting with the simplest and progressing to the most complex. The following video provides an excellent overview of these levels.

Let's summarize the key takeaways and add some nuance.

Level 1: Fixed-Size Chunking (Character and Token)

This is the "naive" approach.

• Character Splitting: Splits text every N characters. It's simple and fast but ignores all semantic and syntactic boundaries. It will often cut words and sentences in half.
• Token Splitting: A slight improvement where you split every N tokens. This is better aligned with how language models process text, as both embedding models and LLMs have token limits. However, it still suffers from the same core problem of arbitrarily breaking logical units.

Impact:

• Context Preservation: Very poor. High risk of fragmenting complete ideas.
• Retrieval Granularity: Arbitrary and often nonsensical. A retrieved chunk might be an incomplete sentence.
• When to use: Rarely in production. It can be useful for quick prototyping or for highly uniform, unstructured data like log files.

Level 2: Recursive Character Text Splitting

This is the go-to strategy for most initial RAG implementations.

It works by attempting to split text along a prioritized list of separators. The default order is typically:

1. Double newlines (\n\n) - tries to keep paragraphs together.
2. Single newlines (\n) - tries to keep lines together.
3. Spaces ( ) - tries to keep words together.
4. Characters ('') - as a final fallback.

The process is "recursive" because if a split by a higher-priority separator (like a paragraph) results in a chunk that is still too large, the algorithm will then try to split that oversized chunk using the next separator in the list (like a sentence).

Impact:

• Context Preservation: Good, especially for prose. It respects the author's intended structure (paragraphs, sentences).
• Retrieval Granularity: More meaningful. Chunks tend to correspond to logical blocks of text.
• Why it's a strong baseline: It offers a great balance of performance and simplicity (the best ROI, as the video notes). Research from vector database companies like Chroma has shown this simple method performs surprisingly well, making it an excellent starting point.

Level 3: Document Structure-Aware Chunking

Your project involves processing PDFs, which are often more than just plain text. They can contain tables, figures, code snippets, and a clear hierarchical structure (chapters, sections, subsections). A robust ingestion pipeline should leverage this structure.

This approach moves beyond treating all text as equal.

• For Code: You can split based on classes or functions, keeping logically related code together.
• For Markdown/HTML: You can split by header levels (#, ##, etc.), creating chunks that correspond to document sections. This also allows you to add the header as metadata to the chunk, giving the LLM more context.
• For Tables & Images: Instead of extracting messy text, you can keep a table's structure (e.g., convert to Markdown or HTML) or generate a textual summary of an image. This preserves information that would otherwise be lost.

Impact:

• Context Preservation: Excellent. It maintains the logical and visual structure of the document.
• Retrieval Granularity: Highly relevant. You can retrieve an entire function, table, or section, which is often exactly what a user needs.
• Relevance to your project: Since you'll be parsing academic papers or textbooks in chemical engineering, which are rich in tables (reaction constants), figures (process diagrams), and equations, this strategy is critically important for building a high-fidelity knowledge base.

Level 4: Semantic Chunking

The strategies so far split text based on its format or structure. Semantic chunking splits text based on its meaning.

The core idea is to find "topic boundaries" in the text. You can do this by embedding each sentence (or small group of sentences) and calculating the similarity between adjacent sentences. A sharp drop in similarity likely indicates a change in topic, making it a good place to create a chunk boundary.

Impact:

• Context Preservation: High semantic coherence. Chunks are formed around consistent topics.
• Retrieval Granularity: Based on meaning, not arbitrary length or formatting, which can lead to very focused chunks.
• Trade-offs: This is computationally expensive, as it requires you to generate embeddings for many small pieces of text just to decide on the final chunk boundaries. The selection of the similarity threshold is also a non-trivial hyperparameter to tune.

Level 5 & Beyond: Experimental and Agentic Strategies

For completeness, it's worth knowing about the cutting edge of chunking research. These methods are powerful but often too complex or expensive for most production systems today.

• Proposition-Based Chunking: Uses an LLM to break down the text into "atomic facts" or propositions. For example, "Einstein, who developed relativity, was born in Germany" becomes two propositions: "Einstein developed relativity" and "Einstein was born in Germany." You then embed these facts. This offers maximum information density but is very expensive and can lose narrative flow.
• Agentic Chunking: Uses an LLM as a "decider." The system iterates through sentences and asks an LLM: "Is this a good place to end the current chunk? Is the thought complete?" This is highly intelligent but also very slow and costly due to the multiple LLM calls per document.

Conclusion

You've now seen how chunking is not a simple preprocessing step, but a critical design choice with a direct impact on the context your RAG system retrieves. The quality of your chunks sets the upper limit on the quality of your retrieval.

Key Takeaways:

• The Problem: Naive fixed-size chunking is destructive because it ignores the semantic structure of text, leading to fragmented context and poor retrieval.
• A Hierarchy of Strategies:
  • Recursive Character Splitting is the recommended starting point, offering a great balance of performance and simplicity.
  • Document Structure-Aware Chunking is essential for complex documents with mixed content (like the PDFs in your project), as it preserves vital structural and multimodal information.
  • Semantic Chunking offers higher coherence by splitting based on meaning, but at a higher computational cost.
• No "One-Size-Fits-All": The best strategy is often a hybrid approach that adapts to the content. For example, using a structure-aware parser to identify tables and code, while applying a recursive or semantic splitter to the remaining prose.

In our next lesson, we will synthesize the topics from this module. Armed with your knowledge of embedding models, evaluation metrics, and chunking strategies, you will make and justify your initial design choices for the project's ingestion pipeline.