# Temperature and Sampling

Decoding controls that shape how the next token is drawn from the predicted distribution — temperature, top-k and top-p.

**Rests on:** Tokenization

**Leads to:** LLM Nondeterminism, Multi-Token Decoding, Sliding Window Sampling

*LLMs and Generative AI · Large Language Models*


## Why learn this

This is the dial that decides whether a chatbot sounds like a careful librarian or a creative storyteller. It directly controls the unpredictability of every AI system that generates text — from ChatGPT to image-captioning tools. Understanding temperature and sampling lets you: make a model give the same answer every time (important for customer support or medical advice), or make it give fresh, varied answers (important for creative writing or brainstorming). It also unlocks your next topics: **why outputs vary** (why you can't always get the same output) and **how words are joined together** (how models generate faster by looking at multiple words at once). Without this, you're just pressing a button and hoping; with it, you're the one choosing the behaviour.


## The intuition

Every time a language model writes the next part of a word or whole word, called a token, it doesn't just pick the most likely candidate. It has a list of all possible next tokens, each with a probability — like a weather forecast: 'sunny 80%, cloudy 15%, rain 5%'. The model then 'rolls a dice' weighted by those percentages. If it rolls sunny most of the time, the text is predictable and safe. But you can change the dice. Temperature is a single number that reshapes those percentages. Low temperature (like 0.1) makes the most likely word almost certain to be chosen — the text becomes repetitive and safe. High temperature (like 1.5) flattens the percentages — even the 'rain' word gets a real chance, so the text becomes surprising, creative, and sometimes nonsense. Top-k and top-p are different knobs: they don't change the percentages, they cut off the list. Top-k says 'only consider the top 10 words, ignore the rest'. Top-p says 'keep adding words from the top until their combined probability reaches 0.9 — so if the top 3 words have 90% probability, ignore all the rest'. Temperature controls the shape of the dice, while top-k and top-p decide how many sides the dice even has.


## An analogy

Think of a language model as a street-food vendor who has to choose one dish to cook next from a menu of a hundred possible dishes. Each dish has a popularity score (probability). A vendor who only ever picks the most popular dish corresponds to temperature 0. A creative vendor wants to surprise customers, so they sometimes pick a less popular dish — that's high temperature. But they don't want to serve something nobody likes (like a word that makes no sense), so they limit the menu: top-k says 'only consider the 10 most popular dishes, pick randomly among those'. Top-p says 'keep the dishes that together cover 90% of customer demand — maybe that's the top 3, maybe the top 20, depending on how popularity is spread'. Temperature is how much you let the vendor deviate from the crowd; top-k and top-p are how many dishes are even on the table. Where the analogy breaks down: a real vendor has taste and experience, but the model has no understanding of the meaning of the words — it's purely a statistical game. Also, temperature doesn't add new knowledge; it just changes how often the model picks rare words, not what it knows.


## Definition

Temperature and sampling are controls that shape how the next token is drawn from the predicted probabilities, where temperature sharpens or flattens those probabilities, while top-k and top-p limit the list of candidates before one is chosen.


## How it connects

You already know that a language model predicts the next token — that's the core of **Large Language Models** in your notes. You also know that **LLM Nondeterminism** means the same prompt can give different outputs — temperature is the main knob that controls how different those outputs can be. Your notes on **Context Window** tell you that the model's memory is only the tokens it sees; temperature decides how it picks the next one from that memory. This topic sits right after **Sliding Window Sampling** (how training examples are made) and **Byte-Pair Encoding** (how words become tokens) — those are about the input side, while temperature is about the output side. It also connects to **Reasoning Models** (they trade latency for accuracy) — and temperature affects how 'careful' the reasoning looks. In your book 'A Common-Sense Guide to AI Engineering', temperature is described as the 'creativity dial' on the API — and top-p as 'nucleus sampling' because it packs the probabilities into a nucleus of words.


## The mathematics

This concept draws on **probability** — the branch of mathematics that deals with how likely things are. You already know probability intuitively from weather forecasts: '80% chance of rain' means out of 100 similar days, it rained on 80. The model does the same: it assigns a probability to each possible next word, and the probabilities always add up to 100% (or 1 if we write them as decimals). 

The key operation here is **sampling**: picking one item from a list, where each item has a chance proportional to its probability. You do this in real life when you roll a weighted dice — a dice with a lead weight on the '6' side comes up 6 more often. Temperature changes the weights on the dice.

Mathematically, temperature is a number you use to reshape the probabilities before you pick a word. The model first produces raw scores for each word. These raw scores are not probabilities yet; they are just numbers that tell us how much the model likes each word. To turn these into probabilities, we follow three steps:

1. **Divide by Temperature**: We take each raw score and divide it by the temperature number. If the temperature is low (like 0.5), the high scores get much bigger relative to the low scores. If the temperature is high (like 2.0), the high scores get smaller relative to the low scores, making them more similar to each other.

2. **Exponentiate**: We raise the mathematical constant e (approximately 2.718) to the power of each divided score. This step exaggerates the differences. A score that is slightly higher becomes a much larger number after this step. A score that is lower becomes a much smaller number.

3. **Normalize**: We add up all these new large numbers. Then, for each word, we divide its specific number by the total sum. This ensures that all the final probabilities add up to exactly 1 (or 100%).

**Top-k** is simple: you just take the k highest-probability words and ignore the rest. You then renormalize the remaining probabilities so they still add up to 1.

**Top-p** is a bit more subtle: you keep adding words to your list, starting from the most likely, until their combined probability reaches a threshold (like 0.9). The number of words you keep varies depending on how spread out the probabilities are. If one word has 90% chance, you keep only that one. If the top ten words each have 10%, you keep all ten.

The mathematics itself is just arithmetic — multiplication, division, addition, and powers — nothing more.


## A worked example

A model predicts the next word after 'The cat sat on the'. The raw scores (logits) for the top 5 words are:
- mat: 2.0
- sofa: 1.0
- floor: 0.5
- bed: 0.5
- table: 0.5

We will calculate the probabilities using a low temperature of 0.5 and a high temperature of 2.0.

**Case 1: Temperature = 0.5 (Sharp)**

Step 1: Divide each raw score by the temperature 0.5.
- mat: 2.0 / 0.5 = 4.0
- sofa: 1.0 / 0.5 = 2.0
- floor: 0.5 / 0.5 = 1.0
- bed: 0.5 / 0.5 = 1.0
- table: 0.5 / 0.5 = 1.0

Step 2: Raise e (approx 2.718) to the power of each result.
- mat: e^4.0 ≈ 54.60
- sofa: e^2.0 ≈ 7.39
- floor: e^1.0 ≈ 2.72
- bed: e^1.0 ≈ 2.72
- table: e^1.0 ≈ 2.72

Step 3: Sum these values to get the total.
Total = 54.60 + 7.39 + 2.72 + 2.72 + 2.72 = 70.15

Step 4: Divide each value by the total to get the probability.
- mat: 54.60 / 70.15 ≈ 0.78 (78%)
- sofa: 7.39 / 70.15 ≈ 0.11 (11%)
- floor: 2.72 / 70.15 ≈ 0.04 (4%)
- bed: 2.72 / 70.15 ≈ 0.04 (4%)
- table: 2.72 / 70.15 ≈ 0.04 (4%)

Result: With low temperature, 'mat' has a very high chance (78%). The output is predictable.

**Case 2: Temperature = 2.0 (Flat)**

Step 1: Divide each raw score by the temperature 2.0.
- mat: 2.0 / 2.0 = 1.0
- sofa: 1.0 / 2.0 = 0.5
- floor: 0.5 / 2.0 = 0.25
- bed: 0.5 / 2.0 = 0.25
- table: 0.5 / 2.0 = 0.25

Step 2: Raise e to the power of each result.
- mat: e^1.0 ≈ 2.72
- sofa: e^0.5 ≈ 1.65
- floor: e^0.25 ≈ 1.28
- bed: e^0.25 ≈ 1.28
- table: e^0.25 ≈ 1.28

Step 3: Sum these values.
Total = 2.72 + 1.65 + 1.28 + 1.28 + 1.28 = 8.21

Step 4: Divide each value by the total.
- mat: 2.72 / 8.21 ≈ 0.33 (33%)
- sofa: 1.65 / 8.21 ≈ 0.20 (20%)
- floor: 1.28 / 8.21 ≈ 0.16 (16%)
- bed: 1.28 / 8.21 ≈ 0.16 (16%)
- table: 1.28 / 8.21 ≈ 0.16 (16%)

Result: With high temperature, 'mat' has only a 33% chance. The other words are much more likely to be picked. The output is varied.


## From scratch, in Python

```python
# Step 1: We have three Python building blocks to use.
# - A list: an ordered collection of values, e.g. [2.0, 1.0, 0.5].
# - A for loop: repeats an action for each item in a list.
# - A function: a reusable block of code with a name.

# Step 2: Our raw scores (logits) from the model for 5 words.
logits = [2.0, 1.0, 0.5, 0.5, 0.5]

# Step 3: Temperature application — divide each logit by T.
def apply_temperature(logits, temperature):
    # Create a new list, one per logit, divided by temperature.
    new_logits = []
    for z in logits:
        new_logits.append(z / temperature)
    return new_logits

# Step 4: Softmax — convert logits to probabilities that sum to 1.
def softmax(logits):
    # First compute e^z for each z (using the math library).
    import math
    exps = []
    for z in logits:
        exps.append(math.exp(z))
    # Sum all exponentials.
    total = 0
    for e in exps:
        total = total + e
    # Divide each exponential by the total.
    probs = []
    for e in exps:
        probs.append(e / total)
    return probs

# Step 5: Top-k — keep only the k highest logits, zero out the rest.
def top_k(logits, k):
    # Make a copy so we don't change the original list.
    filtered = logits[:]
    # Find the k-th largest value (sort descending, take the k-th).
    sorted_logits = sorted(logits, reverse=True)
    if len(sorted_logits) > k:
        threshold = sorted_logits[k-1]
        for i in range(len(filtered)):
            if filtered[i] < threshold:
                filtered[i] = -999999  # zero out after softmax
    return filtered

# Step 6: Top-p — keep the smallest set of words whose cumulative probability reaches p.
def top_p(logits, p):
    # First apply softmax to get probabilities.
    probs = softmax(logits)
    # Sort words by probability (highest first), but remember original indices.
    indexed = list(enumerate(probs))  # e.g. [(0, 0.7), (1, 0.2), ...]
    indexed.sort(key=lambda x: x[1], reverse=True)
    # Keep adding until cumulative probability >= p.
    cum = 0
    keep_indices = []
    for idx, prob in indexed:
        cum = cum + prob
        keep_indices.append(idx)
        if cum >= p:
            break
    # Zero out logits not in keep_indices.
    filtered = [-999999] * len(logits)
    for idx in keep_indices:
        filtered[idx] = logits[idx]
    return filtered

# Step 7: Finally, sampling — pick a word based on probabilities.
def sample(probs):
    import random
    # Random number between 0 and 1.
    r = random.random()
    cum = 0
    for i, prob in enumerate(probs):
        cum = cum + prob
        if r < cum:
            return i
    return len(probs) - 1

# Step 8: Put it all together.
words = ["mat", "sofa", "floor", "bed", "table"]
logits = [2.0, 1.0, 0.5, 0.5, 0.5]

temp_logits = apply_temperature(logits, 0.5)
final_logits = top_p(top_k(temp_logits, 3), 0.9)
probs = softmax(final_logits)
chosen_index = sample(probs)
print("Probabilities:", probs)
print("Chosen word:", words[chosen_index])
```


## The same thing, with a framework

```python
In practice, you don't write this yourself. With a library like Hugging Face Transformers, you call: outputs = model.generate(input_ids, max_new_tokens=50, temperature=0.7, top_k=50, top_p=0.9). Underneath, that library is doing exactly what we wrote above: it takes the logits from the model, applies temperature by dividing, applies top-k by zeroing out small logits, applies top-p by keeping only the top nucleus, converts to probabilities (softmax), and samples one token.
```


## What the code does

We started with a list of logits, one per candidate word. The `apply_temperature` function loops through each logit and divides by the temperature — that's the sharpening or flattening. `softmax` turns those scores into probabilities: it applies the exponential function (making big scores huge, small scores tiny), sums them, and divides each by the sum, so they all add up to 1. `top_k` keeps the highest k by sorting a copy of the list, finding the threshold, and replacing anything below it with an extremely negative number (which after softmax becomes effectively zero probability). `top_p` first converts to probabilities, sorts them descending, and keeps taking words until cumulative probability reaches p. Finally, `sample` does the actual 'dice roll': it generates a random number between 0 and 1 and walks through the probabilities, adding them up, until it crosses the random number — that's the weighted random choice.


## Common misconceptions

**Temperature 0 makes the model deterministic** — If I set temperature to 0, I'll always get the exact same output.

Even greedy decoding can vary due to parallel execution (batching), floating-point arithmetic differences across GPUs, and non-deterministic operations in the model itself, as noted in your **LLM Nondeterminism** notes. In fact: Temperature 0 makes the model always pick the most likely token, but the model's internal computations can still produce tiny differences that snowball, so deterministic output is never fully guaranteed.

**Higher temperature makes the model smarter** — Turning up the temperature makes the AI more intelligent or more creative in a good way.

Creativity is not intelligence. High temperature makes the model pick less probable words, which may sound creative but just as often produces nonsense. The model's knowledge stays the same. In fact: Temperature changes the randomness of the output, not the model's knowledge or reasoning ability. High temperature increases variety and risk; low temperature increases precision and repetition.

**Top-k and top-p are the same thing** — Top-k and top-p both just limit the number of words, so they're interchangeable.

Top-k uses a fixed number (e.g., always 10 words) regardless of the probabilities. Top-p uses an adaptive number — it keeps adding words until their combined probability reaches the threshold, so it might keep 3 words in one case and 30 in another. In fact: Top-k is a fixed-size filter; top-p is a probability-mass filter. They behave very differently when probabilities are evenly spread versus concentrated on one word.


## In short

Temperature is the creativity dial: low makes the model repeat the most likely token (safe, consistent), high makes it sample rarer tokens (varied, surprising). Top-k and top-p are safety valves that trim the list of candidates before sampling. In practice, choose low temperature for extraction and exact tasks, high temperature for open-ended creative tasks, and tune top-p to cut off nonsense. You'll end up setting both every time you call a language model API.


## Where this is used

This is the most practically important dial in AI development. **Large language models** — like ChatGPT, Claude, or Llama — all use temperature and top-p as API parameters. When you set temperature=0, you get the most likely token every time (greedy decoding) — but your notes on **LLM Nondeterminism** remind you that even temperature 0 isn't fully deterministic because of parallel processing and hardware. Developers working on **customer support chatbots** set temperature low (0.2–0.3) so answers are consistent and factual; they also set top_p low (0.5–0.7) to cut off unlikely words entirely. For **creative writing tools**, temperature goes up to 0.8–1.2, letting the model surprise the user. For **code generation**, a low temperature (0.1–0.3) is common because code must be exact, while high temperature leads to 'hallucinated' API calls. In **small models running on phones** (like a voice assistant on a smart speaker), temperature is often locked to a low value because the model is smaller and more prone to errors; the developer also sets top_k low to avoid rare tokens that might not even be in the on-device vocabulary. In **retrieval-augmented generation (RAG)**, where you've retrieved a document and ask the model to answer from it, temperature is kept low (0.1–0.2) — you want the model to extract facts, not embellish. In **agentic systems** that plan and call tools, temperature is a balance: you want the plan to be creative enough to adapt, but the tool calls (like function names or JSON arguments) must be exact. So developers often use a moderate temperature (0.5–0.7) for the 'planning' step and a low temperature (0.1) for the actual tool call generation. In **everyday applied ML** — like predictive text on your phone, autocomplete in search, or translation — temperature is set based on the cost of a mistake: translation for legal documents gets 0.1, while recipe suggestions get 0.9. The actual developer workflow: in code, you pass parameters like chat.completions.create(model='gpt-4', messages=..., temperature=0.7, top_p=0.9) — that's it. The library calls the model, the model produces logits for all possible next tokens (say 50,000), the sampling code we wrote applies temperature and top-p, and the model returns one token. The hardest part in practice is **picking the right temperature**: too low, and the model repeats itself and sounds robotic; too high, and it generates plausible-sounding nonsense. Many developers build a small evaluation set — 50 prompts — and run the model at several temperature settings, checking which setting gives the best answers according to their metric (like factual accuracy or a human preference score). What breaks when done badly: setting temperature too high for a financial chatbot could literally recommend a bad investment; setting it too low for a creative brief could make the output a cliché. Temperature is also a lever in **LLM-as-judge** setups — if you're using a model to evaluate other outputs, you set its temperature to 0 because you want consistent grading.


---

Read this on the web: https://basavapurushottam.com/learn-ai/temperature-and-sampling
