Diagnosing Causes of Repetitive LLM Output

Hello again. Last time, you calculated how overlap increases duplicated chunk text—and why that duplication reaches Claude only when the overlapping chunks are retrieved together. This lesson takes the next step: when an answer repeats a phrase or seems to “stutter,” how do you tell whether the cause is the context supplied to the model, the chunks retrieval selected, or the way the model generated its answer?

The practical goal is a diagnosis you can reproduce in a Java and Spring Boot RAG application, rather than a reflexive change to the overlap setting.

First, locate the repetition

Suppose a document says, “The sensor alarm triggers at 8 mg/L.” That sentence appears at the end of one chunk and, because of overlap, at the start of the next. Both chunks rank highly for a question about the alarm. The answer comes back:

The alarm triggers at 8 mg/L. The alarm triggers at 8 mg/L.

The repeated answer is real, but the explanation is not yet established. The model might have echoed duplicated evidence. Alternatively, retrieval might have selected two copies from different documents, even without adjacent-chunk overlap. Or the supplied context might contain the sentence just once while generation falls into a repetition loop.

Start by comparing three things from the same request:

1. The final answer: Is repetition present in the raw model response, or only after your application displays or concatenates it?
2. The assembled prompt: Which retrieved passages were actually sent, in what order? Are instructions or conversation history repeated too?
3. The retrieval results: Which source documents and chunks were selected? Are repeated passages adjacent overlapping chunks, copies from separate sources, or repeated selections of one chunk?

This distinction matters because duplicate context is a risk factor, not a deterministic instruction to repeat. Conversely, an answer can stutter even when every retrieved passage is unique.

Input and output token counts can tell you that a call was unusually large or long. They cannot, by themselves, identify which sentences were duplicated.

Why generation can repeat without duplicate chunks

An LLM generates a token while taking its previously generated tokens into account. Once it begins a repetitive pattern, that pattern becomes part of the context for subsequent tokens. Restrictive decoding can make the same continuation more likely again. A prompt with repeated instructions or an unfinished list can also invite a similarly shaped continuation.

Read the relevant portions of Sebastian Raschka’s Why LLMs get stuck in repetition loops. They explain the generation-side mechanisms you need to keep separate from chunking.

For a Java-code answer, repeating a class name, field name, or punctuation may be necessary. Do not classify every repeated token as stuttering. Look instead for unwanted repetition of a sentence, clause, list item, or answer structure.

Run a controlled comparison in Spring Boot

The cleanest first test changes the supplied chunks, not the model settings. Capture the ordered list of segments selected by retrieval for one question. Make a second list that removes redundant evidence while retaining at least one chunk containing the fact needed to answer. Keep the question, instructions, model, and generation settings unchanged.

The following small probe assumes your Spring application already has a configured LangChain4j ChatModel bean. It accepts retrieved segments, not all segments produced at ingestion:

package com.example.rag.diagnostics;

import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.chat.ChatModel;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.stream.IntStream;

@Service
public class StutterProbe {

    private final ChatModel model;

    public StutterProbe(ChatModel model) {
        this.model = model;
    }

    public ProbeResult compare(
            String question,
            List<TextSegment> retrieved,
            List<TextSegment> reduced) {

        String originalAnswer = model.chat(prompt(question, retrieved));
        String reducedAnswer = model.chat(prompt(question, reduced));

        // Separate check: can this setup repeat without retrieved context?
        String independentAnswer = model.chat(
                "In one sentence, explain what an HTTP status code indicates.");

        return new ProbeResult(
                originalAnswer, reducedAnswer, independentAnswer);
    }

    private String prompt(String question, List<TextSegment> evidence) {
        String context = IntStream.range(0, evidence.size())
                .mapToObj(i -> "[chunk %d]\n%s"
                        .formatted(i + 1, evidence.get(i).text()))
                .reduce((left, right) -> left + "\n\n" + right)
                .orElse("(no evidence)");

        return """
                Answer using only the supplied evidence.
                If it is insufficient, say so.

                Evidence:
                %s

                Question: %s
                """.formatted(context, question);
    }

    public record ProbeResult(
            String originalAnswer,
            String reducedAnswer,
            String independentAnswer) {}
}

For this comparison, retrieved must be a snapshot of what your retrieval stage actually selected. Construct reduced deliberately: if two neighboring chunks both contain the alarm sentence, remove one only if the other still contains enough information to answer. Otherwise, a worse answer could merely mean you removed essential evidence.

The calls cost tokens and may vary from run to run. If the original answer repeats and the reduced answer does not, that supports a duplicate-context explanation; one pair of outputs does not prove it. Repeat the comparison a few times, and inspect the actual prompts. The unrelated question is a separate check for repetition without retrieved chunks—it is not a test of whether your RAG answer remains factually supported.

If reducing the chunks helps, identify why those chunks were present. Adjacent segments sharing a boundary point toward overlap. Duplicate text from independent sources, repeated selection of the same segment, or several near-identical high-ranked results point toward retrieval or source-data behavior. Lowering ingestion overlap might address the first case but leave the others unchanged.

Only after checking the selected context should you experiment with a different overlap and re-index the document. An overlap of zero is a useful diagnostic comparison, not automatically a good production setting: as you saw last time, it may separate a fact from the context needed to interpret it.

Change generation settings one at a time

If the raw model output loops despite nonduplicated evidence—or on unrelated prompts—investigate generation. Temperature affects how concentrated token choices are; top-p and top-k restrict the candidate pool in different ways. Availability and permitted combinations depend on the model and provider, so check what your configured Claude integration actually supports before trying a setting.

Watch these short parts of Gary Explains’ The Secret Controls for your LLM: Temperature, Top-K, Top-P, etc. They give enough intuition to make a controlled settings change without treating a single value as a cure.

For a settings test, hold the exact same assembled prompt fixed and change one supported parameter at a time. Run more than once if sampling is involved. If repetition changes while the prompt stays identical, generation settings are implicated. If an answer merely stops midway, also check its output-token limit and stopping behavior: truncation is not the same symptom as a repetition loop.

Anti-repetition penalties are provider-specific and can suppress legitimate reuse of Java identifiers, quoted evidence, or factual terms. Treat them as an option to evaluate after you know the failure mode, not as a substitute for inspecting retrieved context.

Key takeaways

To diagnose stuttering, follow the evidence at the point where it enters the model: compare the raw answer, the assembled prompt, and the selected chunks. Then test a version with redundant retrieved evidence removed while preserving the answerable fact. Adjacent duplicated passages suggest overlap; duplicates from elsewhere suggest retrieval or source-data behavior; loops without duplicate context warrant a generation-side investigation. No single observation proves a cause, so change one factor at a time.

Next, you will move beyond fixed-size windows to recursive splitting, which tries paragraph, sentence, word, and token boundaries to preserve more natural units within a token budget.