Text Chunking Strategies

Welcome to the second module of our course! In the previous module, we laid the groundwork by exploring advanced prompting, context management, and the high-level strategic choice between Retrieval-Augmented Generation (RAG) and fine-tuning. We concluded that RAG is often the best approach for tasks requiring up-to-date, external knowledge. Now, it's time to roll up our sleeves and start building a RAG system.

This lesson marks our first practical step into the RAG pipeline: document ingestion. Before we can retrieve relevant information, we must first process and store our knowledge base. A critical part of this process is chunking—breaking down large documents into smaller, manageable pieces.

Our goal today is to understand and implement two foundational chunking strategies: fixed-size character splitting and the more intelligent recursive character splitting. Mastering these techniques is fundamental to building an effective retrieval system that can accurately find and use information.

Why Do We Need to Chunk Documents?

Before diving into implementations, let's establish why chunking is non-negotiable in a RAG system. As a software engineer, you can think of this as a preprocessing step to optimize data for a specialized downstream service—the LLM. There are two primary technical constraints that make chunking essential.

As the video explains, the two main drivers for chunking are:

1. Context Window Limits: Every LLM has a maximum context window—a finite limit on the number of tokens it can process in a single request. You simply cannot feed a 100-page document into a model with an 8,000-token context limit. Chunking ensures the data we pass to the model fits within this constraint.
2. Signal-to-Noise Ratio for Retrieval: This is the more subtle but arguably more important reason. When a user asks a question, we want to retrieve only the most relevant snippets of text. If our chunks are too large and contain multiple, unrelated topics, we might retrieve a chunk where the relevant sentence is buried among pages of irrelevant "noise." This noise can confuse the LLM and degrade the quality of its answer. Smaller, semantically focused chunks lead to better retrieval and, consequently, better generation.

Strategy 1: Fixed-Size Character Splitting

The most straightforward way to chunk a document is fixed-size character splitting. The logic is simple: you define a chunk_size (e.g., 1000 characters) and slice the document into pieces of that length.

This method also introduces a key parameter: chunk_overlap. This defines a number of characters that are repeated at the end of one chunk and the beginning of the next. The purpose of overlap is to preserve context that might otherwise be lost at an arbitrary split point. For example, if a sentence is cut in half between two chunks, the overlap ensures the full sentence is available in at least one of them.

However, this simplicity comes at a significant cost. Since this method is completely unaware of the text's structure or meaning, it often creates nonsensical splits right in the middle of words or sentences.

While you would rarely use this strategy in a production system due to its semantic ignorance, understanding it is crucial for appreciating the more advanced methods.

Here is a basic implementation using Python and the popular langchain library, which you will encounter frequently in AI engineering.

from langchain.text_splitter import CharacterTextSplitter




# Some text to split
with open('sample_document.txt', 'r') as file:
    sample_text = file.read()




# Initialize the splitter
# This will split the text every 1000 characters with a 200 character overlap
char_splitter = CharacterTextSplitter(
    separator="\n\n", # A placeholder, but CharacterTextSplitter splits by character count first
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len # Uses python's default len() to measure size
)

chunks = char_splitter.create_documents([sample_text])




# Let's inspect the first chunk
print(f"First chunk has {len(chunks[0].page_content)} characters.")



# print(chunks[0].page_content)




# If the first chunk's end was abrupt, the overlap ensures context is carried over.
# The second chunk will start 200 characters before the first one ended.
if len(chunks) > 1:
    print(f"Second chunk has {len(chunks[1].page_content)} characters.")



    # print(chunks[1].page_content)

The key takeaway is that fixed-size splitting is a blunt instrument. It solves the context window problem but often harms the signal-to-noise ratio by breaking semantic units. We need a more intelligent approach.

Strategy 2: Recursive Character Splitting

This is the workhorse of chunking strategies and the recommended starting point for most RAG applications. Recursive character splitting is a significant improvement because it attempts to respect the document's structure.

Instead of a single, arbitrary character count, this method uses a prioritized list of separators. A common hierarchy for prose is:

1. \n\n (double newline, for paragraphs)
2. \n (single newline, for lines)
3. . (period and space, for sentences)
4. (space, for words)
5. `` (empty string, for characters as a last resort)

The "recursive" process works as follows:

1. It first tries to split the entire document by the highest-priority separator (e.g., \n\n).
2. It then looks at the resulting chunks. If any of them are still larger than the specified chunk_size, it takes that specific chunk and recursively splits it using the next separator in the list (e.g., \n).
3. This process continues down the hierarchy until all chunks are within the desired size.

This approach is powerful because it tries its best to keep semantically related content, like paragraphs or sentences, together.

Implementation with LangChain

While understanding the from-scratch implementation is valuable, in practice, you'll use a library. langchain provides a ready-made RecursiveCharacterTextSplitter.

from langchain.text_splitter import RecursiveCharacterTextSplitter




# Some text to split
with open('sample_document.txt', 'r') as file:
    sample_text = file.read()




# Initialize the recursive splitter
# It will use a default list of separators: ["\n\n", "\n", " ", ""]
recursive_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len
)

chunks = recursive_splitter.create_documents([sample_text])

print(f"Created {len(chunks)} chunks using the recursive splitter.")



# Check the end of a chunk - it's much more likely to be a natural boundary
# print(chunks[0].page_content[-100:]) 

The code is nearly identical to the fixed-size example, but the underlying logic is far more robust. The splitter intelligently navigates the text to find the best possible split points.

Choosing a Baseline Strategy

For any new project, RecursiveCharacterTextSplitter should be your default choice. It offers the best balance of simplicity and performance. The decision of which chunking strategy to use depends heavily on the type of data you are indexing.

As the chart suggests, for "Clean prose / Markdown," a recursive splitter with overlap is the standard starting point. You would then evaluate its performance and decide if more advanced techniques are necessary. We will explore some of those more advanced techniques in upcoming lessons.

An important detail for choosing chunk_size and chunk_overlap is that these are typically measured in characters by default. However, LLMs operate on tokens. In a previous lesson, we discussed calculating token counts. For more precise control, you can provide a token-counting function to the splitter's length_function parameter. A good rule of thumb for overlap is 10-20% of the chunk size.

Conclusion

In this lesson, we took our first practical step in building a RAG pipeline by tackling document chunking. You now understand not only why we chunk but also how to implement two fundamental strategies.

Key Takeaways:

• Chunking is essential for fitting documents into an LLM's context window and for improving the signal-to-noise ratio during retrieval.
• Fixed-Size Character Splitting is the most basic method. It's simple but semantically naive, often breaking text at awkward points.
• Recursive Character Splitting is the industry-standard baseline. It uses a hierarchy of separators to intelligently split text along natural structural boundaries like paragraphs and sentences.
• For any new project dealing with prose or semi-structured text, start with a RecursiveCharacterTextSplitter. Tune the chunk_size and chunk_overlap based on your specific documents and the target LLM's context window.

The strategies we've discussed are based on the physical structure of the text. In our next lesson, we will go a step further and explore semantic chunking, a more advanced technique that uses language models themselves to identify and create chunks based on their meaning, not just their punctuation.