Comparing Claude Token Counts Across Text Types and Estimation Methods

Good to see the pipeline vocabulary put to use. Previously, we separated tokenization from splitting, chunking, and embedding: the tokenizer measures and encodes text, while the splitter decides which text belongs in a retrieval segment.

This lesson makes tokenization concrete. You will build a small Spring Boot measurement utility that compares English prose, Greek prose, Java code, and Markdown using a Claude-oriented TokenCountEstimator. The important outcome is not memorizing a universal “characters per token” conversion—it is learning to measure representative content with the tokenizer that governs your application’s budget.

Tokens are model-specific units, not words

A token is a piece of text recognized by a model’s vocabulary. It might be a whole common word, part of a long identifier, punctuation, whitespace-associated text, or a short fragment of a non-English word.

The familiar English approximation—about four characters or three quarters of a word per token—is a planning heuristic, not a contract. It can be broadly useful for ordinary English prose, but it becomes unreliable when your source material contains Greek, code, identifiers, Markdown, tables, JSON, URLs, or substantial punctuation.

For Claude-oriented work, distinguish three levels of confidence:

1. A rough rule of thumb helps during initial design.
2. A TokenCountEstimator measurement helps configure and test your Java application.
3. Anthropic’s token-counting endpoint, using the actual request payload, is the authority when an exact production limit or cost calculation matters.

The LangChain4j interface lists AnthropicTokenCountEstimator among its implementations. Use the Anthropic estimator for a Claude-based system; an OpenAI estimator is not interchangeable merely because both return an integer called “token count.”

Why text type changes the result

A useful comparison controls as much as possible. Keep each sample short at first, use a fixed model configuration, and record both the source text and its measured count. Then compare the count to transparent, deliberately crude baselines.

The comparison between English and Portuguese below makes the general point visually: similar meaning and almost the same number of characters can yield different token totals. It is an illustration of one sentence, not a conversion factor for every language. In particular, it should not be used to infer an exact Greek multiplier.

Greek is especially worth measuring from your own corpus. A marine-science knowledge base might include station names, place names, scientific terminology, Greek notes, English abstracts, and numerical observations. That mixture will not behave like generic English documentation.

A word count is also not a neutral baseline. For prose, “word” can reasonably mean a sequence separated by whitespace. For Java, public, Optional<Reading>, stationId, and ); can each become a “word” under that definition, although they plainly play different roles in tokenization. Use the word count only to expose how poor a proxy it may be.

The ratios you will calculate are:

Use Unicode code points, not String.length(), for the character baseline. Java’s String.length() counts UTF-16 code units, which can differ from human-visible characters for some Unicode text. Even code-point counting is still a measurement convention rather than a perfect count of visual graphemes, but it is a much better comparison for multilingual text.

A reusable Spring Boot measurement service

The following service deliberately separates:

• a Claude-compatible token estimate supplied by LangChain4j;
• a Unicode code-point count;
• a simple whitespace-word count.

It does not pretend the latter two are model limits. They are comparison columns that help you spot why a 2,000-character sample may be inexpensive in one format and unexpectedly large in another.

package com.example.rag.tokens;

import dev.langchain4j.model.TokenCountEstimator;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;

import java.util.regex.Pattern;

@Service
public class TokenMeasurementService {

    private static final Pattern NON_WHITESPACE = Pattern.compile("\\S+");

    private final TokenCountEstimator claudeTokenCountEstimator;

    public TokenMeasurementService(
            @Qualifier("claudeTokenCountEstimator")
            TokenCountEstimator claudeTokenCountEstimator) {

        this.claudeTokenCountEstimator = claudeTokenCountEstimator;
    }

    public TokenMeasurement measure(String label, String text) {
        int codePoints = text.codePointCount(0, text.length());

        int whitespaceWords = (int) NON_WHITESPACE
                .matcher(text)
                .results()
                .count();

        int estimatedClaudeTokens =
                claudeTokenCountEstimator.estimateTokenCountInText(text);

        return new TokenMeasurement(
                label,
                codePoints,
                whitespaceWords,
                estimatedClaudeTokens
        );
    }

    public record TokenMeasurement(
            String label,
            int codePoints,
            int whitespaceWords,
            int estimatedClaudeTokens) {

        public double charactersPerToken() {
            if (estimatedClaudeTokens == 0) {
                return 0.0;
            }
            return (double) codePoints / estimatedClaudeTokens;
        }

        public double wordsPerToken() {
            if (estimatedClaudeTokens == 0) {
                return 0.0;
            }
            return (double) whitespaceWords / estimatedClaudeTokens;
        }
    }
}

The qualifier is intentional. A RAG application can have several estimators—perhaps one for Claude chat generation and another related to a different model integration. Naming this bean claudeTokenCountEstimator makes the dependency explicit and avoids quietly using the wrong tokenizer.

Configure that named bean with the Anthropic implementation available in your installed LangChain4j version. The API lists AnthropicTokenCountEstimator, but the exact builder or constructor details can vary with LangChain4j releases. Keep the concrete construction in configuration; the service should depend only on the stable TokenCountEstimator interface.

Measure four representative samples

Put the following test in src/test/java. It produces a compact report you can copy into a spreadsheet, a test log, or an internal architecture note.

package com.example.rag.tokens;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import java.util.List;
import java.util.Locale;

@SpringBootTest
class TokenMeasurementServiceTest {

    @Autowired
    private TokenMeasurementService tokenMeasurementService;

    @Test
    void compareContentTypes() {
        List<Sample> samples = List.of(
                new Sample(
                        "English prose",
                        """
                        The tide gauge reported a gradual rise in sea level.
                        The monitoring service sent an alert after three readings
                        exceeded the configured threshold.
                        """
                ),
                new Sample(
                        "Greek prose",
                        """
                        Ο σταθμός μέτρησε σταδιακή άνοδο της στάθμης της θάλασσας.
                        Η υπηρεσία παρακολούθησης έστειλε ειδοποίηση όταν τρεις
                        μετρήσεις ξεπέρασαν το καθορισμένο όριο.
                        """
                ),
                new Sample(
                        "Java code",
                        """
                        public Optional<Reading> latestReading(UUID stationId) {
                            return repository
                                    .findTopByStationIdOrderByRecordedAtDesc(stationId);
                        }
                        """
                ),
                new Sample(
                        "Markdown with Java fence",
                        """
                        # Station alert

                        The **latest reading** exceeded the configured threshold.

                        ```java
                        alertService.raise(stationId);
                        ```
                        """
                ),
                new Sample(
                        "Markdown rendered as plain text",
                        """
                        Station alert

                        The latest reading exceeded the configured threshold.

                        alertService.raise(stationId);
                        """
                )
        );

        for (Sample sample : samples) {
            TokenMeasurementService.TokenMeasurement measurement =
                    tokenMeasurementService.measure(sample.label(), sample.text());

            System.out.printf(
                    Locale.ROOT,
                    "%n%s%ncode points: %d | whitespace words: %d | "
                            + "estimated Claude tokens: %d | chars per token: %.2f | "
                            + "words per token: %.2f%n",
                    measurement.label(),
                    measurement.codePoints(),
                    measurement.whitespaceWords(),
                    measurement.estimatedClaudeTokens(),
                    measurement.charactersPerToken(),
                    measurement.wordsPerToken()
            );
        }
    }

    private record Sample(String label, String text) {
    }
}

There are two Markdown samples on purpose. They isolate the effect of formatting:

• The first contains a heading marker, bold delimiters, blank lines, fenced-code markers, and Java syntax.
• The second carries approximately the same human-readable message without Markdown markup or a code fence.

Do not expect all Markdown to be “expensive.” A short heading and a few blank lines can make little difference. A document full of tables, embedded JSON, links, HTML fragments, or code blocks is a different case.

When running the test, record the following for every result:

Avoid turning one run into a permanent company rule such as “Greek costs exactly twice as much as English.” A two-sentence sample is a diagnostic probe, not a statistical study. For a more realistic result, later measure a representative set of actual source documents: Greek prose, English manuals, Java configuration and source files, and Markdown runbooks.

Interpret differences rather than merely collecting numbers

Suppose your Java sample has fewer code points than the English sample but a similar or larger token count. That is not an error in the service. It tells you that character count concealed the syntactic density of source code: punctuation, generic type delimiters, camel-case pieces, and identifiers influence the tokenizer.

Likewise, if the Greek sample has a less favorable characters-per-token ratio than the English sample, it does not mean Greek is intrinsically less efficient in every context. It means this model’s token vocabulary represented these particular strings in that way. Names, accents, transliterated terms, and domain vocabulary can all change the result.

resource_id="bbc13"
sections="0,1"
title="Claude Token Counter: Count Tokens & API Cost — Calcxi"
estimated_time="6 minutes"
---
Read this short third-party explanation for its useful distinction between common words, unusual terms, code, and non-English text. Treat its browser counter as an estimate, not as a replacement for Anthropic’s own counting service.
---
In “What is a token in Claude?”, read <tf start="Claude does not read your prompt word by word." end="but not always.">the opening explanation</tf>, then continue through the short paragraphs on rare words, code, and other languages. Next, in “How accurate is this Claude token counter?”, read <tf start="Anthropic has never published Claude's tokenizer." end="use that.">the accuracy discussion</tf>. Focus on the practical conclusion: browser-side approximations are suitable for rough sizing, whereas production-critical counts should use Anthropic’s token-counting endpoint.

One additional distinction matters in a chat application:

• estimateTokenCountInText(text) measures the document or snippet alone.
• estimateTokenCountInMessage(message) measures a message representation.
• estimateTokenCountInMessages(messages) is closer to a conversational input, but the full request can also include a system prompt, tool definitions, retrieved content, and provider-specific request structure.

Therefore, do not add a guessed number of “wrapper tokens” to each document segment. When an application is close to Claude’s context limit, count the actual request structure through the provider’s supported mechanism and leave operational headroom.

Key takeaways

• Tokens are model-specific text units, not words or characters.
• For a Claude-based LangChain4j application, measure content with an Anthropic-compatible TokenCountEstimator, not an estimator designed for another model family.
• English prose heuristics are useful only as rough planning estimates.
• Greek prose, Java syntax and identifiers, Markdown markers, and fenced code can all diverge substantially from character- or word-based expectations.
• Use Unicode code points and whitespace words as transparent comparison baselines, not as token-budget substitutes.
• For cost-sensitive or context-limit-sensitive production requests, validate with Anthropic’s token-counting endpoint using the complete request payload.

Next, you will use these measurements to implement fixed-size, token-aware splitting with configurable overlap in a LangChain4j Spring Boot application.