Semantic Chunking with Sentence-Aware Splitting

Hello! Welcome back to our journey into building Retrieval-Augmented Generation systems.

In our last lesson, we established a solid foundation for document ingestion by implementing recursive character splitting. This is a robust strategy that respects the physical structure of a document, like paragraphs and sentences. However, its main limitation is that it's "semantically blind"—it doesn't understand the meaning of the text it's splitting.

Today, we will address that limitation head-on. Our goal is to implement semantic chunking by leveraging sentence-aware splitting techniques. This approach moves beyond structural boundaries and aims to partition documents based on topical shifts. By creating chunks that are internally coherent, we can significantly improve the quality of information we retrieve, which is a crucial step toward building a high-performing RAG system.

The Core Idea: Splitting by Meaning

The fundamental difference between recursive and semantic chunking lies in the logic used to create boundaries. While recursive splitting uses a hierarchy of characters (\n\n, \n, . ), semantic chunking uses the conceptual similarity between sentences.

The general algorithm is as follows:

1. First, break the document down into its constituent sentences.
2. Next, generate a numerical representation, or embedding, for each sentence. An embedding is a vector of numbers that captures the semantic meaning of the text.
3. Then, calculate the similarity between the embeddings of adjacent sentences. A common metric for this is cosine similarity.
4. Finally, if the similarity score between two consecutive sentences drops below a certain threshold, it signifies a topic change. This is where we insert a "breakpoint" and start a new chunk.

This method ensures that sentences discussing the same topic are grouped together, regardless of their length or where they appear in a paragraph.

Visualizing Semantic Breakpoints

Before we dive into writing code, it's helpful to build a strong mental model of how this process works. What does a "semantic shift" actually look like in the data?

The visualization in the video is powerful. The flat areas of the graph represent a sequence of sentences on a consistent topic, while the sharp peaks represent the transition to a new topic. Our goal is to turn this visual intuition into a working implementation.

Implementing an Embedding-Based Semantic Chunker

Now, let's translate this concept into Python. We will build a SemanticChunker class that encapsulates the logic we've just discussed. Given your background as a software engineer, organizing this logic into a class will feel natural and provides a reusable, configurable component for our RAG pipeline.

Let's break down the key components of the SemanticChunker class from the article, which you will use as the basis for your own implementation:

• __init__(...): The constructor sets up the chunker. It loads a sentence-transformer model (like all-MiniLM-L6-v2) which will generate our embeddings. It also defines key parameters you can tune:

  • similarity_threshold: This is the crucial value that determines sensitivity to topic shifts. A lower value creates fewer, larger chunks; a higher value creates more, smaller chunks. This is analogous to the percentile threshold in the video.
  • min_chunk_size / max_chunk_size: These are guardrails to prevent chunks that are nonsensically small or still too large.

• split_into_sentences(text): This is the first step. The provided implementation uses regular expressions. In a production system, you might use a more robust NLP library like nltk or spaCy for more accurate sentence boundary detection.

• compute_embeddings(sentences): This method takes a list of sentences and passes them to the sentence-transformer model, which returns the numerical embedding vectors. We will explore embeddings in much more detail in our next lesson. For now, think of this as a function that converts text into a meaningful numerical representation.

• find_breakpoints(sentences, embeddings): This is the core logic. It iterates through the sentence embeddings and calculates the cosine_similarity between each adjacent pair (i and i+1). If the similarity is less than the similarity_threshold, it records that position as a breakpoint.

• chunk(text): This is the main public method. It orchestrates the entire process: splitting into sentences, computing embeddings, finding breakpoints, and finally, grouping the sentences between those breakpoints into the final chunks. It also includes logic to merge chunks that are too small, which adds robustness.

By implementing this class, you will have a functional semantic chunker that you can integrate into the RAG pipeline we are building.

The Broader Landscape of Semantic Chunking

The embedding-based method we just implemented is the most common and practical approach. However, it's not the only one. For an AI Engineer, knowing the landscape of possibilities is crucial for making informed architectural decisions.

Let's briefly explore two other families of semantic chunking.

As the article explains, the main categories are:

1. Embedding-based: This is what we've focused on. It's a great balance of performance and computational cost. Libraries like langchain and llamaindex, as well as the custom implementation we just studied, use this approach.
2. Model-based: This approach leverages the pre-training tasks of models like BERT. For example, the "Next Sentence Prediction" (NSP) task in BERT was designed to determine if two sentences logically follow each other. One could use this signal to decide where to break chunks. These methods are less common in practice for chunking but are conceptually interesting.
3. LLM-based (Propositions): This is the most advanced and expensive method. It uses a powerful LLM (like GPT-4) with a carefully crafted prompt to break down a document into a series of "propositions"—atomic, self-contained statements of fact. For example, the sentence "The CEO, who founded the company in 2010, announced record profits" might be broken into two propositions: "The CEO announced record profits" and "The CEO founded the company in 2010." This creates extremely precise, factual chunks but comes at a high cost due to the required LLM calls.

For a production system, you would almost always start with an embedding-based approach due to its favorable cost-performance profile.

Conclusion

In this lesson, we have moved beyond simple structural splitting and learned how to divide documents based on their semantic content. This is a critical optimization for any serious RAG system.

Key Takeaways:

• Semantic chunking groups related sentences together, creating chunks that are more topically coherent than those from structural methods.
• The most common implementation is embedding-based. It involves generating embeddings for sentences and splitting chunks where the semantic similarity between adjacent sentences drops below a threshold.
• You can implement this from scratch, as we've studied, or use libraries like langchain and llamaindex which have pre-built semantic chunkers.
• Tuning the similarity threshold is key to controlling the granularity of your chunks.
• More advanced (and expensive) methods exist, such as using an LLM to generate atomic propositions, which represents the cutting edge of retrieval granularity.

The concept of embeddings was central to our work today. We used them as a tool to measure meaning, but we didn't look too closely at the tool itself. In our next lesson, we will do just that. We'll explore how to generate vector embeddings for our text chunks using different models and how to index them into a specialized vector database, preparing them for efficient retrieval.