Understanding Chunk Size and Overlap Trade-offs

Hello again. In the previous lesson, you built a Spring Boot DocumentSplitter with a maximum estimated-token size and a configurable overlap cap. Now the question is what those settings cost and what they protect. You will calculate how many chunks a document produces, how much text is repeated, and how much room retrieved chunks could take in a Claude prompt.

We will first use an idealized token window, where sizes and overlaps are exact. Then we will return to your actual splitter, where the estimator and text boundaries make those numbers less tidy.

Why repeat text at a boundary?

Imagine a document in which the last line of one chunk identifies a monitoring station, while the first line of the next gives its alert condition. Retrieving either chunk alone might leave out part of the fact. Overlap copies text from the end of the first chunk into the beginning of the second, giving a boundary-spanning passage a chance to appear together.

Read the “Chunk Overlap” and “The Precision-Context Trade-off” sections of Document Chunking: Optimizing RAG Retrieval Pipelines. The first explains the boundary benefit and its duplication cost; the second explains why simply making chunks larger is not always the answer. Its mathematical notation may not render clearly in the provided text, so use the worked calculations below for the formula.

The key limitation is that overlap protects only what it actually repeats. It does not guarantee that a long explanation spanning a boundary will fit in one chunk, or that retrieval will select the chunk containing the whole explanation.

Calculate chunk count and duplicated text

Let $L$ be a document’s length in tokens, $C$ the maximum tokens in a chunk, and $O$ the exact number of tokens repeated between neighboring chunks. For this calculation, assume $0\le O<C$, exact token positions, and chunks filled to capacity except possibly the last.

The first chunk covers $C$ tokens. Each later chunk advances through $C-O$ new tokens. Call that advance the stride:

For a nonempty document, the number of chunks is

An empty document produces zero chunks. The ceiling matters: even one remaining token needs another chunk.

Suppose $L=1000$, $C=400$, and $O=60$. The stride is $340$, so

Using half-open token ranges, the chunks cover positions $[0,400)$, $[340,740)$, and $[680,1000)$. Their lengths are 400, 400, and 320. The two repeated regions are 60 tokens each.

If every boundary repeats exactly $O$ tokens, duplicated token occurrences and the total amount of chunk text are

Here, $D=120$ and $T=1120$. The index still represents a 1000-token source; it stores 1120 token occurrences across its chunks.

Keeping the same hypothetical document makes the effect of each setting visible:

Notice the threshold effect: increasing overlap from 0 to 60 adds duplicated text but does not add a chunk in this example. Increasing it to 200 does. Reducing the chunk maximum to 200 creates more, narrower retrieval units even though its overlap is much smaller.

These equations describe exact, fixed-stride token windows. Your previous lesson’s splitter instead cuts text at Unicode code-point boundaries and uses a token estimator. Its overlap is an upper bound and may be reduced to leave room for new text. Use the equations to predict a configuration’s behavior, then measure the emitted segments rather than assuming the prediction is an exact count.

Separate index cost from prompt cost

More or larger chunks affect two different stages. At ingestion, duplicated text can increase the amount of segment text to embed and store. At generation, text uses Claude’s context only if those segments are retrieved and included in the request. Indexing 600 duplicated token occurrences does not mean every question sends 600 extra tokens to Claude.

For example, with the 400-token maximum and 60-token overlap, retrieving all three chunks from the worked example would supply 1120 token occurrences while covering 1000 unique source-token positions. Retrieving just the first and third would supply $400+320=720$ occurrences with no shared source positions between them. Which chunks retrieval selects determines the duplication in a particular prompt.

A simple capacity bound for $K$ retrieved chunks is $KC$ tokens of chunk text. With three chunks of at most 400 tokens, that is at most 1200. It is not a complete request budget: instructions, the question, conversation history, formatting, and room for the answer also matter.

Watch this short portion of Choosing the right Chunk Size for RAG by Matt Williams. It shows a retrieval design that limits the total material forwarded to a model rather than fixing only the number of chunks. The demonstration budgets in words; your Claude-oriented application should make its own decisions using token estimates.

There is a further retrieval cost: if a query matches text in an overlap region, two adjacent chunks may both rank highly. They can occupy two retrieval slots while contributing much of the same evidence. That is a possible effect of overlap, not a guarantee that the model will repeat itself.

Check the boundary, not just the totals

A configuration can look inexpensive in a table yet cut through the fact you need. In the 400/60 example, suppose a station identifier occupies positions near 390–400 and its alert condition begins at 400–410. The second chunk begins at 340, so it contains both pieces. With zero overlap, neither neighboring chunk necessarily contains the complete fact.

Now suppose the essential setup begins near position 315 and its conclusion lies beyond 400. A 60-token overlap starts the second chunk at 340: it cannot carry that earlier setup forward. More overlap might help, but it would also increase duplication and might change the number of chunks.

This suggests two complementary checks:

• Numerical: How many chunks and repeated token occurrences does the configuration produce?
• Semantic: Do likely questions have their supporting facts together in at least one retrievable chunk?

Neither check replaces the other. Making chunks very small can produce focused matches that lack their explanation; making them very large can mix several topics into a less focused match.

Measure your Spring Boot splitter

You can audit the splitter from the previous lesson without changing its implementation. This service records how many segments it produced and estimates the source and each emitted segment with the same Claude-oriented estimator:

package com.example.rag.chunking;

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.DocumentSplitter;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.TokenCountEstimator;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;

import java.util.List;

@Service
public class ChunkBudgetAuditService {

    private final DocumentSplitter splitter;
    private final TokenCountEstimator estimator;

    public ChunkBudgetAuditService(
            DocumentSplitter splitter,
            @Qualifier("claudeTokenCountEstimator")
            TokenCountEstimator estimator) {
        this.splitter = splitter;
        this.estimator = estimator;
    }

    public Audit inspect(String sourceText) {
        List<TextSegment> segments =
                splitter.split(Document.from(sourceText));

        long summedSegmentEstimates = segments.stream()
                .mapToLong(segment ->
                        estimator.estimateTokenCountInText(segment.text()))
                .sum();

        int largestSegmentEstimate = segments.stream()
                .mapToInt(segment ->
                        estimator.estimateTokenCountInText(segment.text()))
                .max()
                .orElse(0);

        return new Audit(
                segments.size(),
                estimator.estimateTokenCountInText(sourceText),
                summedSegmentEstimates,
                largestSegmentEstimate
        );
    }

    public record Audit(
            int chunkCount,
            int sourceEstimate,
            long summedSegmentEstimates,
            int largestSegmentEstimate
    ) {}
}

Compare reports after changing rag.chunking.max-tokens and rag.chunking.overlap-tokens in application.yml. The largest segment estimate checks the configured cap; the count and sum show how much segment material the application produced. Inspect adjacent TextSegment.text() values too, especially around a fact that crosses a cut.

Do not label summedSegmentEstimates - sourceEstimate as an exact duplicate-token count. A tokenizer or estimator can count a phrase differently when it appears alone, next to different text, or inside the full document. For exact duplicated source text, the most reliable audit would additionally record each segment’s start and end offsets in the original document and measure which ranges recur. The report above is a useful budget signal, not a substitute for those offsets.

Key takeaways

Chunk size controls the maximum context within one retrieval unit; overlap trades additional duplicated text for better coverage at boundaries. For ideal fixed windows, the stride $C-O$ determines chunk count, and each additional boundary contributes $O$ repeated token occurrences. In your Spring Boot splitter, treat those calculations as predictions and check the actual segments, estimates, and boundary-spanning facts.

Next, you will use these distinctions to investigate “stuttering”: whether repeated output relates to duplicated retrieved context, the retrieval selection itself, or generation settings.