Recommended resources
The freely available resources this lesson will direct you to.
Hello, and welcome. This course builds toward a private document assistant that runs locally: one that can find information in PDFs and photos, answer questions about it, and show where an answer came from. Before choosing models or writing an API, it helps to understand what a language model actually receives and produces.
In this lesson, you’ll follow a question about a receipt through four ideas: tokenization, the context window, inference, and probabilistic generation. Together, they explain why a document assistant cannot simply “read everything,” why a local response may take time, and why a fluent answer still needs to be checked against its source.
Text becomes tokens before the model sees it
Imagine asking, “What was the total on my bicycle repair receipt?” The model does not receive that sentence as an indivisible string. A tokenizer breaks it into pieces called tokens and maps those pieces to numerical token IDs. The model processes the IDs; a tokenizer later turns generated IDs back into readable text.
A token is not necessarily a word. A common word might be one token, while an unusual name might be several. Spaces, punctuation, dates, file names, and amounts can also be split in ways that are not obvious from their appearance. Different models may use different tokenizers, so the same receipt excerpt can have different token counts.
Read the following parts of Hugging Face’s Tokenizers lesson for the mechanics. The comparison of word, character, and subword approaches explains why many tokenizers use pieces smaller than whole words.
Hugging Face’s Tokenizers lesson explains how text becomes model input and how generated token IDs become readable text again.
In “Word-based,” start with the tokenizer comparison. Continue through “Character-based” and stop at the end of the main “Subword tokenization” explanation, before “And more!” Focus on the trade-off between vocabulary size and the number of tokens needed to represent text. Then, under “Encoding,” read text to IDs. Under “Decoding,” read from the explanation that begins “from vocabulary indices” through IDs to text. You can skip the library API examples for now.
For the assistant you’ll build, tokenization has a practical consequence: document length in pages or words does not tell you exactly how much model input it will consume. A page dense with account numbers, dates, or imperfectly extracted text may use more tokens than a clean page of ordinary prose. A token count is therefore a better basis for deciding how much document text to include in a model request.
Token IDs are also not a document index. Tokenization makes text processable by the language model; finding the right receipt among many files is a separate problem that the course will address later.
The context window is a shared token budget
A model’s context window is the maximum number of tokens it can consider together during generation. Think of it as a capacity limit for a request, not as a promise that the model will reliably use every detail within that limit.
The following local-model demonstration makes the limit visible. It uses LM Studio and a machine with a separate graphics card. Your course’s planned Apple Silicon setup uses unified memory, shared by the CPU and GPU, rather than that machine’s separate VRAM; the underlying lesson is still relevant: larger configured contexts consume more memory and can slow inference.
Why LLMs get dumb (Context Windows Explained)
In “Why LLMs get dumb (Context Windows Explained),” NetworkChuck demonstrates token counting, a configured context limit, and the hardware cost of increasing it.
Watch the token demonstration in LM Studio. Notice that the same text can receive different token counts with different tokenizers, and watch what happens when conversation text exceeds the configured window. Then watch the larger-context demonstration, focusing on which other material consumes context and why the model’s advertised maximum may be impractical on available hardware. Treat the conversation-memory comparison as an analogy, not as literal human memory.
For a document assistant, the window must accommodate more than the user’s question. It may include application instructions, recent conversation, excerpts from files, file-and-page labels, and the answer as it is generated. In simplified form:
Suppose you configure an 8,192-token window. Instructions and recent conversation use 900 tokens, and five retrieved excerpts use roughly 1,100 tokens each. That is about 6,400 input tokens. Reserving 900 for the answer and 600 as a margin brings the planned total to 7,900—already close to the limit. Headers, citations, and an unexpectedly long excerpt could consume what remains.
If the request exceeds the configured limit, the application must handle it somehow: reject it, shorten material, or omit older content. Content that is no longer supplied is not available for the model to consult. Keeping a document on disk does not automatically put it in the model’s context window. This is one reason a useful assistant selects relevant excerpts instead of attaching an entire document library to every question.
A larger window can fit more evidence, but it is not a free upgrade. It can increase memory use and the time spent processing the prompt. Nor does fitting a detail guarantee that the model will use it correctly. Later lessons will address how to select and check evidence; for now, the key distinction is between information stored in files and information actually present in the current model request.
Inference has a reading phase and a writing phase
Inference means using an already-trained model to produce an output. The assistant is not retraining the model each time you ask about a receipt. It supplies a prompt, the model processes it, and the model generates new tokens.
Two phases help explain the delay you experience:
- During prefill, the model processes the input prompt—including any supplied document excerpts—before it can produce the first output token.
- During decode, it generates subsequent output tokens one at a time, taking its prior output into account. Many runtimes reuse intermediate calculations rather than recomputing the entire prompt for every new token.
Here, decode phase means the model’s token-by-token generation phase. That is related to, but distinct from, the tokenizer’s decoding of token IDs into readable text.
LLM Inference Explained: Prefill vs Decode and Why Latency Matters
“LLM Inference Explained: Prefill vs Decode and Why Latency Matters” by Ready Tensor separates the work done before the first generated token from the work of continuing the response.
Watch prefill for the prompt-processing stage. Skip ahead to decode and caching; focus on why later tokens need not entail reprocessing the whole input. Finally, watch output length for the effect of asking the model to generate more tokens. The examples illustrate tendencies, not fixed speeds for every model or machine.
This distinction matters when you assess a local assistant. Supplying ten long pages but requesting a one-sentence answer can make time to first token the noticeable delay. Supplying a short excerpt but requesting a lengthy report can make total generation time the larger cost. Time to first token also includes possible overhead outside prefill—such as model loading or application processing—so it is a useful observed metric, not a pure measurement of one model operation.
Streaming a response to the React client can display tokens as they arrive. It improves perceived responsiveness, but it does not make the model generate the complete answer instantaneously. On a local machine, model size, prompt length, output length, and available memory all affect the experience. You will measure those trade-offs with a real local model later in this module.
A likely next token is not a verified fact
At each generation step, the model assigns scores to possible next tokens. A selection method turns those scores into the next token, and the process repeats. This is probabilistic generation: the output is built from successive choices conditioned on the prompt and on what has already been generated.
The settings used to choose tokens affect behavior. Greedy selection takes the highest-scoring next token each time. Sampling can choose among plausible alternatives; temperature controls how strongly it favors the highest-scoring ones. Lower-temperature or greedy output is often more consistent, while more sampling can produce more variation. Neither setting checks whether a receipt actually says the amount the model writes. Even the most likely continuation can be wrong.
Consider two similar files. A March 2025 bicycle-repair receipt says $86.40, while an older receipt says $68.40. If the assistant receives only the older receipt’s excerpt—or receives no receipt text at all—it may still produce a confident-sounding amount. Its ability to complete a sentence is not evidence that it inspected the right file. Repeating the request at a different temperature cannot repair missing source evidence.
A dependable document assistant therefore needs to distinguish generated wording from established facts. For an amount, the relevant excerpt should actually contain the amount; the response should identify the supporting file and page; and if the supplied evidence is insufficient, the assistant should say so. These are application safeguards around generation, not properties guaranteed by generation itself.
Putting the four ideas together
Return to the question, “What was the total on my bicycle repair receipt from March 2025?” Before the model can answer, the application must locate useful document text and place it in the request. The tokenizer determines how many tokens the question, instructions, and excerpt consume. The configured context window limits how much of that material—and how much answer space—can be present at once.
The model then performs inference: it processes the prompt during prefill and generates the answer token by token. Its token choices can yield a clear sentence such as “The total was $86.40,” but clarity alone does not establish correctness. The application still needs to check that $86.40 appears in the appropriate receipt and preserve a reference to its source.
The central lesson is that a local model is a bounded, resource-consuming text generator, not a database of the files on your computer. Tokenization sets the units of its input and output; the context window limits what can be supplied at once; inference determines much of the latency; and probabilistic generation makes evidence checks essential. Next, you’ll distinguish document search from answer generation—and see where retrieval, tools, and agent behavior fit.