Understanding Tokenization, Text Splitting, Chunking, and Embeddings in LangChain4j RAG

Welcome. This first module builds the vocabulary needed to reason about a RAG system without treating every preprocessing step as “chunking.” In a LangChain4j application, a source document is turned into searchable pieces through several distinct operations; each has a different job, different input and output, and different failure modes.

By the end of this lesson, you should be able to look at a Spring Boot ingestion configuration and identify exactly where tokenization, text splitting, chunking, and embedding occur. This distinction is the foundation for choosing sensible chunk sizes and diagnosing retrieval quality later in the course.

The big picture: RAG has ingestion and retrieval

RAG means Retrieval-Augmented Generation. Rather than attaching an entire handbook, PDF collection, or database export to every LLM request, the application retrieves a few relevant passages and supplies those passages as context for the answer.

The system has two main phases:

1. Ingestion prepares your knowledge base before users ask questions. It loads documents, splits them, embeds the resulting segments, and stores them.
2. Retrieval and generation runs when a user asks a question. It finds relevant segments, adds their text to the prompt, and asks the chat model to answer.

The diagram is deliberately simple, but it conveys an important fact: segments and embeddings are not the same thing. A segment is readable text. An embedding is a numerical representation used to locate potentially relevant text.

Four concepts that are often conflated

Consider a Markdown document from an internal engineering handbook:

# Deployment rollback

If the canary error rate exceeds 2%, stop the rollout.

```bash
kubectl rollout undo deployment/payments-api

Notify the on-call engineer and attach the incident ID.

Here is what each concept means.

| Concept | Input | Output | Main purpose |
|---|---|---|---|
| **Tokenization** | Characters | Model-specific tokens | Lets a model process text and enforce token limits |
| **Text splitting** | A `Document` | Smaller textual units | Finds candidate boundaries such as paragraphs or sentences |
| **Chunking** | Split units | Retrieval-ready `TextSegment` objects | Groups content into suitably sized pieces, optionally with overlap |
| **Embedding** | A text segment | Numerical vector | Enables semantic similarity search |

### Tokenization: how a model sees text

A tokenizer converts characters into a sequence of tokens, which are internal units represented by integer IDs. Tokens are not reliably equivalent to words or characters.

For example:

- A common English word may be one token, or several.
- A Greek word can use a different number of tokens than a similarly long English word.
- Java identifiers such as `paymentProcessingConfiguration` may be split into several token pieces.
- Markdown symbols, code fences, indentation, URLs, and punctuation also affect token count.

The exact result depends on the tokenizer used by a particular model family. Therefore:

- “500 characters” is not the same budget as “500 tokens.”
- “500 words” is not a safe proxy for “500 tokens.”
- A tokenizer does **not** understand that a paragraph is a coherent retrieval unit. It only encodes text into model-processable pieces.

Tokenization is an accounting and representation mechanism. It matters for context-window limits, latency, and cost, but it does not itself decide where a document should be split.

### Text splitting: finding useful boundaries

A **text splitter** divides a `Document` according to textual structure. Depending on the splitter, the initial units might be:

- paragraphs,
- lines,
- sentences,
- words,
- characters, or
- matches for a regular expression.

A paragraph splitter sees blank-line boundaries. A sentence splitter attempts sentence boundaries. A recursive splitter starts with coarse structure and falls back to finer boundaries if a unit is too large.

This is a structural operation. Its aim is to avoid awkward fragments when possible, such as separating a sentence from its condition or detaching a heading from its content.

### Chunking: assembling retrieval units

A **chunk** is the retrieval-sized piece you decide to embed and store. In LangChain4j, this is represented by a `TextSegment`.

A splitter often performs both jobs:

1. It breaks the document into natural units, such as paragraphs.
2. It combines as many units as will fit under a size limit into a `TextSegment`.

So, in ordinary RAG discussion, people often say “split the document into chunks.” That shorthand is fine, but implementation work benefits from distinguishing:

- **splitting strategy**: where boundaries are preferred;
- **chunk size**: the maximum size of each stored segment;
- **overlap**: deliberately repeated boundary content between neighboring segments.

For example, a chunk may include the heading `Deployment rollback`, the error-rate condition, and the rollback command. That chunk has enough local context for retrieval. A separate chunk might begin with the last complete sentence from the first, if overlap is configured.

The details of fixed-size chunking and overlap are the focus of the next module. For now, retain the key point: **overlap copies text into multiple chunks; tokenization merely counts or encodes the text.**


```grasp
{
  "type": "exercise",
  "id": "c1653209-2872-4e39-90e5-1e5ac02adabd"
}

How LangChain4j models splitting and chunks

LangChain4j represents the loaded source as a Document and each retrieval-sized result as a TextSegment.

A TextSegment contains:

• the segment’s text;
• copied metadata from the source Document;
• an index metadata value that identifies its order within that source.

That metadata matters. If an answer is based on a segment, metadata can tell you the originating file, heading, page, user scope, or chunk index. The vector alone cannot provide that human-readable provenance.

resource_id="ba637"
sections="2"
title="Document Splitter — LangChain4j RAG Tutorial"
estimated_time="6 minutes"
---
Read LangChain4j’s description of the `DocumentSplitter` contract. This is the direct bridge between the concepts above and the Java objects you will configure in a Spring Boot application.
---
In the “Document Splitter” section, begin at the numbered list introduced by <tf start="They all work as follows:" end="and so on.">the splitter workflow</tf>. Follow the remaining steps in the section as well. Pay particular attention to the difference between initial units, such as paragraphs or sentences, and the final `TextSegment` objects produced under a size limit.

LangChain4j provides splitters for paragraphs, lines, sentences, words, characters, and regular expressions. It also provides a recursive splitter for generic text. The recursive version first tries paragraph boundaries; when a paragraph is too large, it can fall back through lines, sentences, words, and finally characters.

This gives you two separate design choices:

A character-based budget is convenient but approximate. A token-based budget is usually more meaningful when your limits are expressed in model tokens. In the next lesson, you will calculate the implications of those choices rather than relying on rules of thumb.

Embedding: turning a chunk into a searchable vector

An embedding model takes text and produces a fixed-length vector of numbers. Vectors are positioned so that texts with related meanings tend to lie closer together under a similarity measure.

For the rollback example, a user might ask:

What should we do if the canary deployment starts failing?

The question does not use the exact words “error rate exceeds 2%” or “rollout undo.” Keyword matching alone might miss the best passage. With embeddings, the application embeds the question, searches for nearby stored vectors, and retrieves the associated text segment.

A crucial distinction:

• The embedding is used for searching.
• The original TextSegment.text() is what gets added to the LLM prompt.
• The chat model generates an answer from the user question plus retrieved segment text.

The vector database does not usually send a row of floating-point values to Claude or another chat model and expect an answer. It returns the original segment, its score, and metadata after similarity search.

A robust system uses a compatible embedding model and configuration both when indexing documents and when embedding user queries. If you replace the embedding model, existing vectors generally need to be regenerated; vectors from unrelated embedding spaces cannot be meaningfully compared.

A Spring Boot configuration, read by responsibility

The following is a conceptual LangChain4j configuration. It uses a token-aware recursive splitter so that segment size is expressed in tokens rather than characters.

@Configuration
class KnowledgeBaseConfiguration {

    @Bean
    DocumentSplitter documentSplitter(TokenCountEstimator tokenCountEstimator) {
        return DocumentSplitters.recursive(
                500,  // maximum segment size in tokens
                50,   // maximum overlap in tokens
                tokenCountEstimator
        );
    }

    @Bean
    EmbeddingStoreIngestor knowledgeBaseIngestor(
            DocumentSplitter documentSplitter,
            EmbeddingModel embeddingModel,
            EmbeddingStore<TextSegment> embeddingStore) {

        return EmbeddingStoreIngestor.builder()
                .documentSplitter(documentSplitter)
                .embeddingModel(embeddingModel)
                .embeddingStore(embeddingStore)
                .build();
    }
}

Read it in this order:

1. TokenCountEstimator measures text using a chosen tokenizer. It is concerned with the size budget, not semantic search.
2. DocumentSplitters.recursive(...) is the text splitting and chunking policy. It tries meaningful boundaries and produces TextSegment objects that stay within the configured token budget.
3. EmbeddingModel converts each final TextSegment into an embedding vector.
4. EmbeddingStore<TextSegment> persists the vector together with the original segment and metadata.
5. EmbeddingStoreIngestor coordinates these steps when a Document is ingested.

A small Spring component can invoke the configured ingestor:

@Component
@RequiredArgsConstructor
class HandbookIngestionService {

    private final EmbeddingStoreIngestor knowledgeBaseIngestor;

    public void ingestHandbook(String markdown) {
        Document document = Document.from(markdown);
        knowledgeBaseIngestor.ingest(document);
    }
}

This service does not tokenize text manually, create vectors manually, or store arbitrary strings in a table. It delegates the pipeline to the configured LangChain4j components.

In a production application, ingestion is often a scheduled job, an administrative action, or a separate service. You normally do not re-ingest an entire knowledge base on every ordinary user chat request. A user’s request should use the already-populated embedding store to retrieve relevant segments.

A practical terminology check

When reading code or discussing a RAG defect, use precise language:

A common misleading statement is: “The embedding model chunks the document.” It does not. The splitter creates the chunks first; the embedding model transforms each chunk into a vector afterward.

Another is: “The tokenizer finds semantic passages.” It does not. A tokenizer encodes text, while the splitter and chunking policy determine what text is grouped together for retrieval.

Key takeaways

A LangChain4j RAG ingestion pipeline has distinct responsibilities:

• Tokenization converts text into model-specific units and supports accurate size limits.
• Text splitting identifies candidate boundaries such as paragraphs and sentences.
• Chunking creates retrieval-ready TextSegment objects within a chosen budget, potentially with overlap.
• Embedding converts each final segment into a numerical vector for similarity search.
• The embedding store preserves both searchable vectors and the original segment text and metadata.
• During a user request, RAG retrieves text associated with similar vectors; the LLM receives that text, not raw vector values.

Next, you will examine token counts directly: English prose, Greek text, Java code, and Markdown can have surprisingly different token-to-character ratios. That measurement will make later chunk-size decisions concrete rather than guesswork.