Welcome to the final lesson of our first module! We've covered a lot of ground, from sending your first prompt to exploring how parameters like temperature can shape an AI's output, and even how to spot and trigger hallucinations. Today, we'll examine a fundamental characteristic of LLMs that underpins many of these behaviors: their non-deterministic nature.
Your learning goal for this session is to run the same prompt multiple times and compare the outputs to observe the non-deterministic nature of LLMs versus traditional code.
If you write a JavaScript function, you expect myFunction('hello') to return the exact same result every single time. LLMs don't work that way. This lesson will show you this in practice and explain why it happens. Understanding this is critical for building reliable AI products, as it changes how we think about testing, consistency, and user experience.
Deterministic vs. Non-Deterministic
First, let's clarify these terms.
- Deterministic systems are predictable. Given the same input, they will always produce the same output. Your calculator is deterministic; 2 + 2 will always equal 4.
- Non-deterministic systems have an element of randomness. The same input can lead to different, though often similar, outputs. Asking a friend for a movie recommendation twice might get you two different answers, even if they're both good suggestions.
Large Language Models fall into the second category. This image provides a great visual summary of the difference:
Your Turn: See Non-Determinism in Action
Let's do a hands-on experiment. You'll need an LLM playground like OpenRouter or a direct interface like Claude.ai.
Experiment:
- Open a new chat in your chosen playground. Use a powerful model like
Claude 3.5 SonnetorGPT-4o. - Make sure the temperature setting is at its default (usually around
0.7or1.0). - Use the following prompt, which connects to your interest in DJing:
Describe the atmosphere of a warehouse rave where Charlotte de Witte is playing a peak-time techno set. - Run the prompt and copy the response to a text editor.
- Now, open another brand new chat window. This is important to ensure the model has no memory of the previous conversation.
- Run the exact same prompt again.
- Compare the two responses.
You will likely notice that while the core ideas are similar (dark, energetic, driving beat), the specific wording, sentence structure, and descriptive details are different. One might focus on the "pounding bass," while the other describes the "hypnotic, strobing lights." This is non-determinism in action.
Why Does This Happen? The Roulette Wheel
The reason for this variability lies in how LLMs generate text. They don't just pick the single "best" next word; they predict a range of possible words and their probabilities. Then, they "sample" from that list.
This process is like spinning a roulette wheel where the size of each slot is determined by the probability of a word.
The following video explains this concept of logits, temperature, and the sampling process using a clear roulette wheel analogy.
Why and How LLMs (AI) Respond Differently to the Same Query
The video "Why and How LLMs (AI) Respond Differently to the Same Query" from the TechnoBotic channel provides an excellent and easy-to-understand explanation of non-determinism.
Watch these two key parts: First, get an overview from the introduction, which defines the concept and uses a great analogy of asking a friend the same question twice. Then, jump to the roulette wheel analogy to see a step-by-step breakdown of how a token is actually chosen. This part is crucial for understanding where the randomness comes from.
Can We Make LLMs Deterministic?
You might be thinking, "What if I set the temperature to 0?" In our previous lesson on parameters, we saw that temperature=0 makes the model always choose the token with the highest probability.
This makes the output mostly deterministic and is a common technique for tasks requiring high consistency, like data extraction or classification. However, it's not a perfect guarantee.
Let's explore why perfect reproducibility is so difficult to achieve.
Why is deterministic output from LLMs nearly impossible? - Unstract
The article "Understanding why deterministic output from LLMs is nearly impossible" from Unstract digs into the deep technical reasons behind non-determinism.
Read the sections titled "Why Temperature Zero Isn’t Really Zero", "Floating-Point Arithmetic: The Silent Culprit", and "The Hardware Lottery". These sections explain that even when the model tries to be deterministic, tiny, almost imperceptible differences in computer hardware calculations can lead to different outputs.
As the article explains, the complex math happening on GPUs isn't perfectly consistent. Tiny rounding differences can occasionally cause the model to pick a different "highest probability" token, leading to a different response. Major providers like OpenAI and Anthropic acknowledge this, stating that their outputs are "mostly" deterministic at temperature=0 but without a 100% guarantee.
Conclusion
This lesson concludes our first module on exploring LLM behavior. You've now seen firsthand the fundamental difference between traditional, deterministic code and the probabilistic nature of AI systems.
Key Takeaways:
- Traditional code is deterministic: same input, same output.
- LLMs are non-deterministic: same input can produce different outputs.
- This is not a bug but a core feature resulting from probabilistic token sampling—the "roulette wheel" selection process.
- Setting
temperature=0makes outputs highly consistent but not perfectly deterministic due to low-level hardware and software variations. - This has major implications for building AI products, affecting everything from testing to ensuring a fair and reliable user experience.
Congratulations on completing Module 1! You've built a solid foundation for understanding how these models work. In our next module, Calling LLM APIs, we'll move from the playground to the code editor. You'll set up your first API keys and learn how to make your first programmatic call to an LLM using your JavaScript skills.