# LLM Nondeterminism

The same prompt can yield different outputs, because sampling is stochastic and even greedy decoding is affected by batching and hardware nondeterminism.

**Rests on:** Softmax and Logits, Temperature and Sampling

**Leads to:** Prompt Iteration

*LLMs and Generative AI · Large Language Models*


## Why learn this

When you use a chatbot or an email assistant powered by a large language model, you may notice that asking the same question twice sometimes gives you different answers. This unpredictability matters for three concrete reasons in your daily work:

1. **Testing is harder**: You cannot write a test that expects one exact answer if the model can return several slightly different ones. You need strategies that accept a range of correct responses.
2. **Downstream systems must be flexible**: If your application sends the model's output to a database or another function, that next step must handle small variations in wording without breaking.
3. **User expectations must be managed**: Users will be confused if they get different answers for the same question. You need to design the product so this variation feels natural or is hidden from them.

Understanding this helps you write robust tests and choose settings that control how much variation occurs.


## The intuition

A large language model does not have a single fixed answer waiting inside it. Instead, it works like this: it looks at your input and calculates a likelihood for every possible next word. For example, if you type "The capital of France is", the model might decide there is a 95% chance the next word is "Paris", a 2% chance it is "a", and a 1% chance it is "the". It also assigns tiny chances to many other words.

When it actually produces the answer, it does not just pick the most likely word every time. Instead, it performs a random selection based on likelihoods. It draws one word from that list of chances. Because "Paris" has the highest chance, it appears most often, but occasionally the model picks "a" or "the" instead.

This randomness is built into how the model operates. When you build larger sentences, these small differences accumulate. If the model picks a slightly different word at position 5, that changes the context for position 6. This can cause the rest of the answer to drift further away from what you might expect.

Even if you try to force the model to be completely predictable by setting its variation setting (often called "temperature") to zero, it is still not perfectly consistent. The computer calculations that happen behind the scenes involve tiny rounding errors because computers approximate real numbers using a limited number of bits. These micro-differences can shift which word is chosen last minute. This is known as floating-point arithmetic imprecision. Because of these computational limits, you cannot rely on identical inputs always producing identical outputs in practice.


## An analogy

Imagine asking a committee of experts the same question: "What is the best way to learn French?" The experts are not robots; they are people who weigh information differently and may feel more like one approach than another on different days. If you ask them today, one might say "immersion," another "daily practice." If you ask again tomorrow, they might give the same answers, or they might swap their priorities based on what they read that morning.

The large language model works similarly but for a mechanical reason: it randomly selects from a list of weighted possibilities each time, rather than because its mind has changed. Unlike the experts, who have fixed knowledge and could theoretically agree to give the same answer if forced, the model’s randomness is fundamental to how it generates text.


## Definition

LLM nondeterminism is the tendency of a language model to produce different outputs for the same input, caused by its method of randomly selecting words based on likelihoods and by tiny rounding errors in computer calculations.

It is best described as: A large language model produces variable outputs for identical inputs because it selects words from a probability distribution rather than choosing a single fixed answer, and because underlying computational approximations introduce minor instabilities.


## How it connects

This concept links directly to **Temperature and Sampling**, which are the controls you use to adjust how much randomness (the random selection based on likelihoods) influences the output. It also connects to **Tokenization** and **Context Window**, because the model makes these random choices one piece of word at a time within a limited memory space. Later, when you study **Retrieval-Augmented Generation** and **AI Agents**, you will see that these systems are designed specifically to handle this variability, often by asking the model multiple times and comparing results to find a reliable answer.


## The mathematics

This concept rests on a branch of mathematics called **probability**. Probability is the study of chance — how likely different outcomes are. You don't need to know any formulas yet, just the idea of a likelihood. When the model predicts the next word, it assigns a number between 0 and 1 to each possible word, where 1 means 'absolutely certain' and 0 means 'never'. These numbers are called **probabilities**. For example, if there are three possible next words — 'apple', 'banana', 'cherry' — the model might say: 'apple' has probability 0.7, 'banana' has probability 0.2, 'cherry' has probability 0.1. These all add up to 1 (since something must come next). The model then **samples** from this distribution: it doesn't simply pick the highest, it picks randomly but weighted by these probabilities. So 'apple' comes out 70% of the time, 'banana' 20%, 'cherry' 10%. This is called **stochastic** (random) **sampling**. The mathematical operation we're using is just adding and multiplying probabilities — no calculus or algebra beyond that. The key idea is that the model's output is not a single fixed value but a random draw from a list of possibilities, each with a different chance.

To understand how the model gets these numbers, imagine it looks at the words you have already typed (the **context**) and asks: "Given what has been said so far, how likely is each possible next word?" Mathematically, we write this as P(next word | context). The vertical bar means "given". So P('apple' | 'I love') is the probability of 'apple' given that you have typed 'I love'.

The model calculates these likelihoods using a mathematical function called **softmax**. You do not need to know how softmax works internally, but you should know what it does: it takes a list of raw scores (called logits) for every possible word and turns them into probabilities that add up to 1. It ensures that no probability is negative and that the total chance is exactly 100%.

Because the model uses this sampling process, it is **non-deterministic**. This means that if you give it the exact same input twice, it might give you two different outputs. It is not making a mistake; it is exploring the space of possible continuations allowed by its training. This randomness is what allows the model to be creative and generate diverse text rather than repeating the single most likely phrase every time.


## A worked example

You are building a simple toy model that predicts the next word after 'I love'. You have a list of three candidate words with probabilities: 'pizza' (0.6), 'coding' (0.3), 'sleeping' (0.1). You run the model twice. Each run, it rolls a metaphorical die weighted by these probabilities.

Step 1: Run 1: The model samples and gets 'pizza'. 
Explanation: The model generates a random number between 0 and 1. If that number falls in the first 60% of the range (0.0 to 0.6), it picks 'pizza'. In this specific run, the random number was 0.45, which is less than 0.6, so 'pizza' is chosen.

Step 2: Run 2: The model samples again and gets 'coding'. 
Explanation: The model generates a new random number. This time, the number was 0.75. This falls in the range for 'coding' (0.6 to 0.9). Even though 'pizza' was more likely, the randomness allowed 'coding' to be selected.

Step 3: Over many runs (say 100), you expect 'pizza' about 60 times, 'coding' about 30, 'sleeping' about 10. 
Explanation: This is the law of large numbers. As you repeat the random process more and more times, the actual frequency of each word appearing will get closer and closer to its assigned probability.

Result: The same prompt 'I love' produces different continuations on different runs because the model samples randomly from the probability distribution, not just picking the top word. This is the essence of LLM nondeterminism.


## From scratch, in Python

```python
# This Python code simulates the nondeterminism of an LLM.
# It does not use any external libraries, just plain Python.

import random

# A list of possible next tokens (simplified).
# A list in Python is an ordered collection of items, like a shopping list.
tokens = ['pizza', 'coding', 'sleeping']

# The probabilities for each token (they must sum to 1).
# A dictionary maps 'keys' (here token names) to values (probabilities).
probabilities = {'pizza': 0.6, 'coding': 0.3, 'sleeping': 0.1}

# Function to sample one token based on the probabilities.
# A function is a block of reusable code; here it takes two arguments.
def sample_next_token(tokens, probs):
    # random.choices picks one item from 'tokens' with weights = 'probs'.
    # The weights are exactly the probabilities we defined.
    return random.choices(tokens, weights=[probs[t] for t in tokens])[0]

# Run the sampling 10 times to see the variation.
for i in range(10):
    token = sample_next_token(tokens, probabilities)
    print(f"Run {i+1}: {token}")
```


## The same thing, with a framework

```python
In real LLM frameworks (like HuggingFace Transformers or OpenAI API), you would call a function like:

from transformers import pipeline

# This loads a small LLM (distilled GPT-2) and generates text.
# The key argument is 'do_sample' — if True, it samples stochastically.
# 'temperature' controls how flat or sharp the probability distribution is.
generator = pipeline('text-generation', model='distilgpt2')
output = generator("I love", max_length=10, do_sample=True, temperature=1.0)

# The underlying mechanism: the model computes logits for all possible next tokens,
# applies softmax to get probabilities, then uses random.choices to pick one.
# 'do_sample=False' would pick the argmax (greedy) every time, but even then
# floating-point non-determinism on GPUs can cause tiny differences.
```


## What the code does

Line by line:
- `import random` brings in Python's random number generator, which we use to simulate the dice roll.
- `tokens = [...]` defines a list of three possible next words.
- `probabilities = {...}` defines a dictionary where each word maps to its probability.
- `def sample_next_token(tokens, probs):` defines a function that takes two arguments. Inside, `random.choices(tokens, weights=[...])` does the sampling — it picks one token from `tokens` with the given weights. The `[0]` at the end extracts the single chosen word from the list of one. 
- The `for i in range(10):` loop runs the sampling 10 times, printing each result. When you run this, you'll see different outputs each time, demonstrating nondeterminism. If you set `do_sample=False` instead, you would always get the top token, but in a real LLM, even that can vary due to hardware.


## Common misconceptions

**Temperature 0 means deterministic** — If I set temperature to 0, I will get the exact same output every time.

Temperature 0 makes the sampling effectively greedy (it always picks the most likely token), which eliminates the randomness from sampling. However, the underlying computation on GPUs involves floating-point rounding that can vary between runs, especially when the model is batched with other requests. Small numerical differences can occasionally change which token is the most likely one, leading to different outputs. In fact: Temperature 0 reduces randomness but does not guarantee determinism. To make outputs more consistent, also set a fixed seed and avoid batching, but even then, slight variation is possible.

**Nondeterminism is a bug** — The model is broken because it gives different answers to the same question.

Nondeterminism is a feature, not a bug. The model is trained to produce a probability distribution over responses, and sampling from that distribution is what enables creativity and adaptability. It is the default behaviour. In fact: Nondeterminism is intentional and often desirable. You can reduce it by using lower temperature or greedy decoding, but you should design your application to handle variation.

**Exact matches are the only way to test** — When testing my LLM app, I should assert that the output exactly equals the expected string.

Because the model is nondeterministic, your test will fail randomly even when the output is correct. Exact matching is too brittle. In fact: Use semantic similarity checks (e.g., embedding cosine similarity above a threshold) or validate that the output contains key facts or matches a schema. This is a standard practice in AI engineering.

**Reproducibility is guaranteed with a seed** — If I set a random seed, I'll always get the same output.

Seeds control the random number generator, but LLM inference is highly parallel and runs on hardware where floating-point operations are not deterministic at the bit level. Batch size, GPU type, and even the order of operations can change the results. In fact: Setting a seed can help in controlled environments, but it is not a guarantee in production. For reproducible results, you may need to use greedy decoding and still accept some variance, or store outputs for auditing.

**All variation is due to sampling** — The only reason I get different answers is because the model samples randomly.

While sampling is the main cause, other factors like hardware nondeterminism (GPU parallel reduction), batching, and even the order of operations in the model implementation can cause variation, even when sampling is turned off. In fact: Variation arises from both sampling and the underlying computational infrastructure. Even with temperature 0, small differences can occur.


## In short

LLM outputs are not fixed. The model predicts a probability distribution over possible next words and then samples from that distribution, which means the same prompt can produce different answers. This is called nondeterminism. You can reduce it by lowering the temperature or using greedy decoding, but complete determinism is impossible due to floating-point and hardware effects. When building applications, you must design tests and downstream systems to tolerate variation — check for semantic equivalence, parse structured outputs, and handle retries. This understanding is foundational for everything you will do with LLMs.


## Where this is used

LLM nondeterminism is not a theoretical curiosity — it affects every real-world application built on top of a large language model. Here is how it plays out across different domains:

**Large language models (the big ones, like GPT-4 or Claude):** When you interact with ChatGPT and ask the same question twice, you often get different answers. This is because the default decoding uses sampling with temperature around 0.7–1.0, which intentionally adds randomness to make the responses more creative and less repetitive. For an application like a creative writing assistant, this is desirable. But for a banking chatbot that needs to give a precise answer about interest rates, the same prompt could give slightly different answers, which is a problem. Developers deal with this by setting temperature to 0 (or a low value like 0.1) for factual tasks, and by using a system prompt that instructs the model to 'be consistent' or 'repeat the answer if asked again'. Even then, because of batching (processing multiple requests at once on a GPU) and floating-point arithmetic, the output can still vary subtly — for example, the model might output '5.9%' one time and '5.9 percent' another.

**Small language models (running on your phone or laptop):** Small models are especially susceptible to nondeterminism because they often run on limited hardware where floating-point precision is lower, and they are more sensitive to prompt formatting. If you are building an app that runs a small model on-device to autocomplete text, you might want deterministic output to avoid embarrassing variations. Developers often use greedy decoding (temperature 0) and set fixed seeds for the random number generator to reduce variance, but even then, the same input may give slightly different outputs on different devices because of hardware differences.

**Retrieval-Augmented Generation (RAG):** In RAG, you first retrieve relevant documents from a database, then feed them to the LLM along with the user's question. Nondeterminism affects both the retrieval and the generation. When you retrieve, the search engine might return slightly different results if you have multiple replicas of the vector database, or if the embedding model is nondeterministic (which it can be, again due to floating-point). Then, when the LLM generates the answer, it might phrase the same fact differently each time. For a RAG-based customer support bot, the answer content should be consistent even if the wording varies. Developers handle this by basing the answer only on the retrieved context and by using low temperature, and they parse the output to extract key information (like order numbers) before presenting to the user, rather than relying on the raw text.

**Agentic systems (AI agents that plan and call tools):** This is where nondeterminism becomes critical. An agent decides what tool to call (e.g., 'search_web' vs 'calculate'), what arguments to pass, and in what order to call multiple tools. If the agent is nondeterministic, it might choose 'search_web' one time and 'calculate' another time for the same user request, leading to completely different outcomes. Even worse, a slight difference in the first step can cause the agent to descend into a completely different plan, so the final answer may be wildly different. Developers mitigate this by using a structured output format (e.g., JSON) and asking the model to output a function call in that format, and by setting the temperature to 0 for the planning steps. But they still have to design the system to handle retries and fallbacks — if the agent fails to call the right tool, it should try again with a different prompt, not just crash.

**Everyday applied machine learning (fraud detection, recommendation systems, etc.):** Most classical ML is deterministic — given the same input, a logistic regression model always gives the same output. But if you incorporate an LLM into your pipeline (for example, to generate a natural language explanation for a fraud alert), that part becomes nondeterministic. The fraud detection score might be deterministic, but the explanation could vary. For compliance reasons, you might want to store the exact explanation given to the customer, so you should record both the prompt and the output. Also, in A/B testing, you need to account for the fact that the LLM's output distribution changes over time (as the model is updated), so comparisons must be done carefully.

**What a developer actually writes:** In practice, a developer controls nondeterminism through three knobs: (1) **temperature** — lower values (0 to 0.2) for factual and deterministic tasks, higher (0.7 to 1.0) for creative tasks; (2) **seed** — some APIs let you set a random seed to make runs reproducible, though this is not a guarantee across different hardware; (3) **sampling parameters** like top-p and top-k — truncating the probability distribution to only the most likely tokens, which reduces the chance of wild outliers. Additionally, developers can use **greedy decoding** (temperature 0 and do_sample=False) which picks the most likely token at each step, which gives the same result most of the time, but floating-point non-determinism can still cause tiny variations. And they write **unit tests** that check semantic equivalence, not exact string match — for example, using an embedding similarity threshold to say 'the model output was acceptable' even if the exact text is different.


---

Read this on the web: https://basavapurushottam.com/learn-ai/llm-nondeterminism
