From Prompt to Prediction: Tokenization and Context-Window Usage

Welcome back. Last lesson distinguished running a model from supplying information through RAG or changing its behavior through fine-tuning. All three eventually depend on inference: the model receives an input and generates an output. Today we’ll follow one short input through that process.

By the end, you should be able to explain how text becomes token IDs, how a model chooses and generates subsequent tokens, and how both the input and the growing answer consume a finite context window. That token budget will matter when you later give a coding assistant repository files or a document assistant retrieved passages.

From text to token IDs

A model does not receive a Python string in the form you read it. A tokenizer first divides the string into pieces called tokens, then maps each piece to an integer ID from its vocabulary. A token might represent a word, part of a word, punctuation, or text that includes a leading space. The rules and vocabulary belong to the particular tokenizer; counting words is therefore not a reliable way to count tokens.

Read the Hugging Face LLM Course’s explanation of this conversion:

Let’s trace an illustrative, invented tokenizer—not a claimed result from any Llama model. Suppose a prompt is The chest is, and this tokenizer produces:

The leading spaces belong to the second and third pieces. The model receives the IDs $[17,284,76]$, not English words. These particular boundaries and numbers are made up; a real tokenizer might split the same prompt differently. It is also important to use the tokenizer associated with the model: an ID has meaning only relative to that model’s vocabulary.

A chat request often contains more than its visible user message. The application may format system, user, and assistant messages with role markers and other special tokens before tokenization. Consequently, the count for a complete chat request can exceed the count you get by tokenizing just The chest is.

How the next token becomes an answer

Once the input IDs are available, the model processes the input and produces a score for every possible next token. These scores, called logits, can be converted into probabilities. The scores are conditional on the sequence so far: change the prompt, and the possible continuations receive different scores.

Continue with our fictional chest example. Imagine the probabilities for the first generated token look like this:

One selection method would take the highest-probability token, locked. Another could sample from the distribution and sometimes choose a different continuation. The selected ID is added to the sequence. The model then predicts again, now conditioned on The chest is locked; perhaps it selects .. A later selection might be an end-of-response token, at which point generation stops. Decoding the generated IDs produces readable text such as locked.

This is next-token prediction, not a one-shot prediction of the whole sentence. The model has learned parameters that produce these scores, but the prompt itself has not changed those parameters. Nor does a high probability certify that a statement is true: an RPG assistant could confidently continue a sentence about a chest without having been given the current game state.

The Hugging Face LLM Course divides inference into two useful phases:

Prefill processes the input sequence so the model can predict its first new token. Decode performs the repeated generation steps, one new token at a time. When the reading says “one word at a time,” interpret that as one token at a time—a token need not be a complete word. Implementations can reuse intermediate calculations from earlier tokens rather than recomputing the entire sequence from scratch at every step; we’ll examine the memory implications later in the course.

The context window is a shared token budget

A model’s context window limits how many tokens can be in the sequence it is using to produce the next token. It is not a word limit, and it is not reserved exclusively for your latest message. Depending on the application, it may need to accommodate instructions, earlier conversation, the current question, selected code or retrieved documents, formatting tokens, and the answer as it grows.

Watch IBM Technology’s What is a Context Window? Unlocking LLM Secrets for a visual explanation:

Here is a concrete budget for a hypothetical local deployment with a configured context limit of 2,048 tokens:

The planned total is $1,380+400=1,780$ tokens, leaving $268$ tokens of headroom. In general, if $C$ is the configured context capacity, $P$ the complete input count, and $G$ the output allowance, plan for:

Now imagine adding another 300-token repository excerpt. The planned total becomes $2,080$, exceeding the limit by $32$ tokens. A sensible response is to choose a shorter or more relevant excerpt, reduce other input, or reduce the reserved answer length. Merely increasing an advertised model context setting may not be possible or practical on the hardware running it.

Notice two subtleties. First, a 700-token excerpt consumes budget even if it turns out not to help answer the question. This is why selecting relevant context matters for coding and RAG assistants. Second, earlier messages are available on a later request only if the application supplies them again, or supplies a representation of them. A model does not independently remember a previous API call. If a limit is exceeded, the model does not universally or reliably “forget the oldest part”; applications and inference servers may reject, truncate, or otherwise handle oversized inputs. For a private assistant, make that behavior an explicit design decision.

A context limit also does not promise that every detail inside it will be used well. It specifies what can fit, not guaranteed comprehension of a long file.

A quick way to inspect a real tokenizer

The toy IDs made the trace easy to see, but real token counts should come from the tokenizer you will actually use. If you already have a local model folder containing tokenizer files and Hugging Face Transformers installed, this optional inspection takes a few minutes:

from transformers import AutoTokenizer

MODEL_DIR = "/path/to/your/local/model"
tokenizer = AutoTokenizer.from_pretrained(
    MODEL_DIR, local_files_only=True
)

text = "The chest is"
ids = tokenizer(text, add_special_tokens=False)["input_ids"]

print("Pieces:", tokenizer.convert_ids_to_tokens(ids))
print("IDs:", ids)
print("Count:", len(ids))
print("Decoded:", tokenizer.decode(ids))

add_special_tokens=False isolates the text itself; it does not measure the full formatted chat request. If your local tokenizer has a chat template, you can inspect that larger count too:

if tokenizer.chat_template:
    messages = [{"role": "user", "content": "The chest is"}]
    chat_ids = tokenizer.apply_chat_template(
        messages, tokenize=True, add_generation_prompt=True
    )
    print("Formatted chat input tokens:", len(chat_ids))

The difference between the two counts is operationally useful. Your future assistants will budget the formatted request actually sent to the model, not just the visible question. If you do not yet have a local model, the conceptual trace and budget above are sufficient for this lesson; you’ll work with model files during the local-inference module.

Key takeaways

A tokenizer splits text into model-specific pieces and maps them to IDs. During inference, the model scores possible next tokens, selects one, and repeats until generation stops; decoding turns output IDs back into text. The initial input is handled during prefill, while generation proceeds through repeated decode steps.

Count the complete formatted input and leave room for generated output within the configured context window. That principle applies whether the extra input is chat history, RPG state, repository code, or retrieved documents. Next, we’ll use context length alongside parameter count, model-card evidence, and license terms to select a Llama-family model for a local deployment.