Welcome to your first lesson in the "Agentic AI with .NET and GCP" course! We're starting at the very beginning of our journey to build intelligent agents. The goal of this lesson is to establish the fundamental principle that powers all Large Language Models (LLMs) like Google's Gemini, which we'll be using extensively.
By the end of this session, you'll be able to explain what a large language model is and, most importantly, how the simple-sounding process of "next-token prediction" is the engine behind their impressive capabilities, from answering questions to writing code. Understanding this core mechanism is crucial, as it influences everything we will do later, including prompt engineering, function calling, and building reliable agents.
Let's begin by demystifying what an LLM actually does.
The Core Idea: A Sophisticated Prediction Machine
At its heart, a Large Language Model is a sophisticated mathematical function trained on a staggering amount of text and code. Its primary, and surprisingly simple, job is to predict what piece of text is most likely to come next, given a sequence of text it has already seen.
To get a quick intuition for this, watch the first part of this short video from 3Blue1Brown, which introduces this core concept.
Large Language Models explained briefly
This video provides a great conceptual overview of LLMs.
Please watch from the beginning until the 1:13 mark. Focus on the analogy of completing a movie script and how the model's job is to predict the next word in a sequence.
As the video explains, the model doesn't "know" anything in the human sense. It calculates probabilities. When you provide an input, or "prompt," the LLM calculates a probability distribution over its entire vocabulary for what the very next piece of text should be.
This diagram illustrates the concept perfectly:
The model takes the input, processes it, and outputs a list of all possible next tokens, each with a calculated probability. For the question "What is 1 + 1?", the token "2" is assigned the highest probability, but other tokens like "3", "car", or "tv" also get assigned a probability, even if it's infinitesimally small. The model's "intelligence" comes from its ability to assign extremely high probabilities to contextually relevant tokens.
From Words to Tokens
You'll notice I've started using the word "token" instead of "word." This is a critical distinction. LLMs don't operate on words but on tokens.
A token is a unit of text that the model processes. It can be a whole word, a part of a word (a subword), a punctuation mark, or even a piece of code. For a deeper look at this, the following article provides a clear, developer-focused explanation.
How LLM Token Prediction Works (A Simple Guide for Developers)
This article from Pavani Singamshetty on Medium is a concise guide for developers. It breaks down the core concepts without getting lost in heavy theory.
Read the introduction and the first section, "What Are Tokens, Really?". This will solidify your understanding of what tokens are and how they relate to the text we write.
Breaking text into tokens allows the model to handle a vast vocabulary, including new or rare words, by constructing them from smaller, known pieces. For a C# developer, you can think of it as the model having its own vocabulary or "character set," which is more complex than just words. This process of converting text into a sequence of tokens is called tokenization.
The image below shows this process for a more natural sentence. The input text "The dog sat on" is broken down into tokens, and the LLM then predicts the probabilities for the next token.
The Generative Loop: Autoregressive Generation
So, if the model only predicts one token at a time, how does it generate entire paragraphs, code blocks, or detailed answers?
The answer is a simple but powerful iterative loop, a process known as autoregressive generation. Here’s how it works:
- The model takes your initial prompt (a sequence of tokens).
- It predicts the most probable next token.
- This newly generated token is appended to the end of the input sequence.
- The model takes this new, slightly longer sequence as its input and repeats the process, predicting the next token.
This "predict, append, repeat" cycle continues until the model generates a special "end of sequence" token or reaches a predefined length limit.
This looping mechanism is fundamental. The following video provides an excellent visual explanation of this entire pipeline, from tokenization to the final loop.
How LLMs Actually Generate Text (Every Dev Should Know This)
This video from the "LearnThatStack" channel is tailored for developers and clearly explains the step-by-step process of text generation.
Watch the segment from probabilities to the loop. Pay close attention to how the output of one step becomes the input for the next. This is analogous to many iterative algorithms you might have implemented in C#.
This autoregressive nature is why generation can sometimes feel slow for long responses; the model isn't just spitting out a pre-written answer. It's performing a full computational pass for every single token it generates. Since you have experience with async/await in C#, you'll see in later lessons that this token-by-token generation is a natural fit for streaming data to a user interface, much like you'd stream chunks of a large file download.
A Bit Deeper: Logits and Probabilities
As a developer, you might be curious about what happens just before the probabilities are calculated. The model's final neural network layer doesn't output probabilities directly. Instead, it produces a raw score for every single token in its vocabulary (which can be over 100,000 tokens). These raw scores are called logits.
Because logits are just numbers and aren't constrained to a neat 0-1 range or to sum to 100%, they are passed through a mathematical function called softmax. The softmax function converts the raw logit scores into a clean probability distribution, where each token's probability is between 0 and 1, and the sum of all token probabilities is 1.
For a more technical summary of this process, the following article is helpful.
How AI Works Under the Hood - LLMs Explained with Code | Nitay Neeman's Website
This article by Nitay Neeman provides a slightly more technical breakdown of the inference pipeline.
Read the section Next-Token Prediction. It formally introduces the terms "logits", "softmax", and "probability distribution". Don't worry about the sampling strategies (temperature, top-k, top-p) for now; we will cover those in detail in a future lesson.
Understanding that the process is probabilistic is key. It explains why even with the same prompt, you can get slightly different answers. It also explains "hallucinations"—when the model generates plausible-sounding but factually incorrect text. It's not lying; it's just following a probable path of token sequences based on the patterns in its training data.
Conclusion
In this introductory lesson, we've unpacked the core mechanism of Large Language Models. Let's summarize the key takeaways:
- LLMs are Next-Token Predictors: Their fundamental task is to calculate the probability of the next token in a sequence.
- Text is Processed as Tokens: LLMs work with tokens (words, subwords, punctuation), not directly with words or characters.
- Generation is an Iterative Loop: Coherent and long-form text is built one token at a time through an autoregressive process (predict, append, repeat).
- Behavior is Probabilistic: The model's output is based on a probability distribution, which explains its creativity as well as its potential for error.
This foundational knowledge is the bedrock upon which all advanced AI agent capabilities are built. The seemingly "intelligent" or "creative" behaviors we see are emergent properties of this surprisingly simple, yet massively scaled, prediction loop.
In our next lesson, we will explore the practical consequences of tokenization by examining tokens, context windows, and how token limits affect prompt and response design. This will be our first step into the engineering challenges of working with LLMs.