Implementing a Token-Aware Fixed-Size Document Splitter with Configurable Overlap

Hello again. In the previous lesson, you measured text with a Claude-oriented token estimator and saw why character and word counts are unreliable substitutes. Now you will put that estimator to work: a Spring Boot application will split a document into token-budgeted segments and repeat a configurable amount of text at each boundary.

This is the ingestion-side implementation. The output is a list of TextSegments that could later be embedded and stored; embedding itself is not part of this lesson.

Define the splitter’s contract

Let maximum size mean the largest token estimate permitted for one segment. Let overlap mean the largest token estimate permitted for text repeated from the end of one segment at the start of the next. For example, a configuration of 400 and 60 means at most 400 estimated tokens per segment and at most 60 estimated tokens of repeated boundary text. It does not promise that every segment contains exactly 400 tokens.

Overlap protects information near a cut. If a station identifier ends one segment and its alert condition begins the next, repeating some text gives retrieval a better chance of returning the two together. It also duplicates text, so we will make the amount explicit rather than repeating an entire segment.

LangChain4j supplies a DocumentSplitter interface, so the Spring application can depend on that interface while we provide our own windowing behavior. We are not calling DocumentSplitters.recursive(...): its paragraph-first behavior is useful, but it is a different splitting strategy that you will implement and examine later.

Implement a token-budgeted window

A TokenCountEstimator can count a string, but it does not give us token positions at which to slice that string. The implementation below therefore considers Unicode code-point boundaries, extending a segment until the next extension would exceed the estimated token budget. It then finds a suffix within the overlap budget to carry into the next segment.

Put this class under src/main/java/com/example/rag/chunking/:

package com.example.rag.chunking;

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.DocumentSplitter;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.TokenCountEstimator;

import java.util.ArrayList;
import java.util.List;
import java.util.Objects;

public final class FixedTokenWindowSplitter implements DocumentSplitter {

    private final TokenCountEstimator estimator;
    private final int maxTokens;
    private final int overlapTokens;

    public FixedTokenWindowSplitter(
            TokenCountEstimator estimator,
            int maxTokens,
            int overlapTokens) {

        this.estimator = Objects.requireNonNull(estimator);
        if (maxTokens <= 0 || overlapTokens < 0 || overlapTokens >= maxTokens) {
            throw new IllegalArgumentException(
                    "Require maxTokens > 0 and 0 <= overlapTokens < maxTokens"
            );
        }
        this.maxTokens = maxTokens;
        this.overlapTokens = overlapTokens;
    }

    @Override
    public List<TextSegment> split(Document document) {
        Objects.requireNonNull(document);

        String text = document.text();
        if (text.isEmpty()) {
            return List.of();
        }

        // Character offsets at Unicode code-point boundaries.
        List<Integer> offsets = new ArrayList<>();
        for (int offset = 0; offset < text.length();
             offset = text.offsetByCodePoints(offset, 1)) {
            offsets.add(offset);
        }
        offsets.add(text.length());

        int last = offsets.size() - 1;
        int start = 0;
        List<TextSegment> segments = new ArrayList<>();

        while (start < last) {
            int end = windowEnd(text, offsets, start, last);

            String chunk = text.substring(offsets.get(start), offsets.get(end));
            segments.add(TextSegment.from(chunk, document.metadata()));

            if (end == last) {
                break;
            }

            start = nextStart(text, offsets, start, end);
        }

        return List.copyOf(segments);
    }

    private int windowEnd(
            String text, List<Integer> offsets, int start, int last) {

        int end = start;

        while (end < last) {
            String candidate =
                    text.substring(offsets.get(start), offsets.get(end + 1));

            if (tokens(candidate) > maxTokens) {
                break;
            }
            end++;
        }

        if (end == start) {
            throw new IllegalArgumentException(
                    "A single code point exceeds maxTokens at index " + start
            );
        }
        return end;
    }

    private int nextStart(
            String text, List<Integer> offsets, int previousStart, int end) {

        int overlapStart = end;

        // Find a suffix that fits the overlap budget. Never repeat
        // the entire preceding segment.
        while (overlapStart - 1 > previousStart) {
            int candidateStart = overlapStart - 1;
            String suffix = text.substring(
                    offsets.get(candidateStart), offsets.get(end));

            if (tokens(suffix) > overlapTokens) {
                break;
            }
            overlapStart = candidateStart;
        }

        // The repeated suffix must leave room for at least one new
        // code point; otherwise reduce the overlap.
        while (overlapStart < end) {
            String overlapPlusNewText = text.substring(
                    offsets.get(overlapStart), offsets.get(end + 1));

            if (tokens(overlapPlusNewText) <= maxTokens) {
                break;
            }
            overlapStart++;
        }

        return overlapStart;
    }

    private int tokens(String text) {
        return estimator.estimateTokenCountInText(text);
    }
}

There are two safeguards worth noticing. First, overlapTokens must be smaller than maxTokens; otherwise overlap could consume a whole window. Second, nextStart reduces the requested overlap if necessary to leave room for new text. The setting is therefore an upper bound, not a guarantee of exactly that many repeated tokens.

This is a compact implementation for learning and modest documents. It repeatedly estimates candidate strings, so it is not optimized for large-scale ingestion. Also, because an estimator supplies counts rather than token offsets, these are estimated-token-bounded text windows, not slices cut at exact Claude token IDs. Each emitted segment is measured against the configured cap.

Wire it into Spring Boot

Keep the two budgets in configuration, where you can tune them without changing splitting code:

# src/main/resources/application.yml
rag:
  chunking:
    max-tokens: 400
    overlap-tokens: 60

Bind the settings and expose the splitter as a Spring bean:

package com.example.rag.chunking;

import dev.langchain4j.data.document.DocumentSplitter;
import dev.langchain4j.model.TokenCountEstimator;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@ConfigurationProperties(prefix = "rag.chunking")
record ChunkingProperties(int maxTokens, int overlapTokens) {
}

@Configuration
@EnableConfigurationProperties(ChunkingProperties.class)
class ChunkingConfiguration {

    @Bean
    DocumentSplitter documentSplitter(
            ChunkingProperties properties,
            @Qualifier("claudeTokenCountEstimator")
            TokenCountEstimator estimator) {

        return new FixedTokenWindowSplitter(
                estimator,
                properties.maxTokens(),
                properties.overlapTokens()
        );
    }
}

This assumes the named claudeTokenCountEstimator bean from your previous lesson is configured in the application. The distinction matters: copying the RAG tutorial’s OpenAI estimator into a Claude pipeline would make the split decisions depend on the wrong token-counting model.

A service can now consume the interface without knowing how the windows are constructed:

package com.example.rag.chunking;

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.DocumentSplitter;
import dev.langchain4j.data.segment.TextSegment;
import org.springframework.stereotype.Service;

import java.util.List;

@Service
public class ChunkPreviewService {

    private final DocumentSplitter splitter;

    public ChunkPreviewService(DocumentSplitter splitter) {
        this.splitter = splitter;
    }

    public List<TextSegment> preview(String sourceText) {
        return splitter.split(Document.from(sourceText));
    }
}

TextSegment.from(chunk, document.metadata()) carries the document’s metadata into the produced segments. If you later add a distinct chunk index or other per-segment metadata, take care not to modify a metadata object shared by multiple segments.

Verify the budget in a Spring test

A useful first test checks the property that matters regardless of whether the source is English, Greek, Java, or Markdown: no emitted segment exceeds the configured estimate.

package com.example.rag.chunking;

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.TokenCountEstimator;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.test.context.SpringBootTest;

import java.util.List;

import static org.junit.jupiter.api.Assertions.assertTrue;

@SpringBootTest
class FixedTokenWindowSplitterTest {

    @Autowired
    @Qualifier("claudeTokenCountEstimator")
    private TokenCountEstimator estimator;

    @Test
    void segmentsStayWithinTheirTokenBudget() {
        var splitter = new FixedTokenWindowSplitter(estimator, 20, 4);

        String text = """
                Station alpha reported rising water.
                The alert threshold was exceeded.
                """.repeat(40);

        List<TextSegment> segments = splitter.split(Document.from(text));

        assertTrue(segments.size() > 1);
        assertTrue(segments.stream().allMatch(segment ->
                !segment.text().isEmpty()
                        && estimator.estimateTokenCountInText(
                                segment.text()) <= 20
        ));
    }
}

Run the same check with a representative Greek document and a Java or Markdown file from your intended corpus. Inspect a few adjacent segment texts as well: you should see a short repeated suffix at their boundary when the overlap budget permits it. With overlapTokens set to zero, adjacent segments should instead advance without intentionally repeating text.

One operational distinction: the segment budget is not the final Claude prompt budget. Later stages may retrieve multiple segments and add instructions, conversation history, and other request content. This splitter controls the size of each ingestion segment; it does not count a complete generation request.

Key takeaways

You now have a DocumentSplitter that uses a Claude-oriented estimator to enforce a per-segment token cap, accepts a configurable overlap cap, preserves Unicode code-point boundaries, and keeps each window moving into new text. Overlap can help preserve boundary context, but it also repeats content and may need to be reduced to fit the next window.

Next, you will quantify the trade-off: how changing chunk size and overlap affects the number of chunks, duplicated text, and eventual context usage.