# Multi-Token Decoding

Generating more than one token per forward pass — speculative decoding and related methods — to reduce the sequential bottleneck.

**Rests on:** Temperature and Sampling, Tokenization

**Leads to:** Latency and Speedups

*LLMs and Generative AI · Large Language Models*


## Why learn this

The single biggest reason chatbot responses take seconds instead of milliseconds is that they generate one word or part of a word at a time, in order — a 500-token answer means 500 sequential steps. Multi-token decoding breaks that bottleneck, and understanding it is the key to making an AI-powered product feel fast. You will use it directly when you build any real-time AI application: a customer-support bot, a coding assistant, or a voice assistant that must reply instantly. It also underlies how reasoning models plan ahead and how systems stay responsive on small hardware. This concept unlocks practical engineering of speed: you will understand what a single step of processing is, why some steps can happen at the same time while others cannot, and how using a smaller, faster model to guess several future words can give you the same answer in half the time.


## The intuition

When you ask a large language model a question, it does not write out its whole answer at once. It produces one small piece — a token, often a word or part of a word — then takes that piece as part of its input to produce the next one, and so on. Each of those steps is called a forward pass: the act of looking at the current text and computing what comes next.

In each step, the model looks at the tokens so far and computes a probability for every possible next token — a likelihood score telling it how likely each candidate word is to be correct — then picks one (usually the most likely). That picked token is added to the sequence, and the whole process repeats. This is sequential: step two cannot start until step one is finished, because the model needs the first token to know what to compute next. There is no way to peek ahead, because the answer is generated, not retrieved.

Multi-token decoding is a set of techniques that try to do the same work in fewer passes. The clever trick is: instead of asking the big, slow model to produce one token at a time, you let a small, fast 'draft' model — a lighter version of an AI designed for speed rather than precision — guess several future tokens in advance. Then you ask the big model to check those guesses all at once.

The big model can verify several guesses in a single forward pass (because checking is parallel — it can score many candidates at once). If the guesses are correct, you are done after one verification, having produced several tokens in one step. If some are wrong, you discard the wrong ones and start over from the last correct guess. The result is exactly the same output you would have gotten the slow way — the big model is the authority, and the draft is just a guess — but you have used fewer sequential passes.


## An analogy

Imagine you are a very thorough, very slow proofreader. Your job is to correct a long manuscript, but you can only look at one word at a time, and after each word you must decide the next word. That is like standard generation. Now imagine you have a quick assistant who is not as accurate but is much faster. The assistant writes down a whole sentence — a guess at what you would write. You then scan the entire sentence at once — because you can check many words in parallel — and you either approve the whole thing or stop at the first mistake. If the assistant is usually right, you save a lot of effort: you correct a whole sentence in one pass instead of word by word. The key is that you are the final authority: if the assistant writes 'the cat sat on the mat' and you agree with every word, you have just produced five tokens in one go. If the assistant writes 'the cat sat on the dog' and you disagree with 'dog', you reject that and everything after it, and you start again from 'on'. The output is identical to what you would have produced on your own, because you are the one who approves every token. The analogy breaks down in one important way: the assistant and the proofreader are not two people — they are two different models.


## Definition

Multi-token decoding is a method for speeding up AI text generation by having a small, fast model propose several future words at once, which a larger, more accurate model then checks together in a single step, accepting the proposal only up to the point where the two models disagree.


## How it connects

This concept sits inside Large Language Models. Your notes already cover Tokenization and Context Window. Tokenization defines what a token is: the small pieces of text that the model reads and writes. Context Window bounds how many tokens the model can consider at once — relevant because both the guessing and checking steps operate within this limit.


## The mathematics

The branch of mathematics this concept draws on is probability — specifically, the idea of a probability distribution over a set of possible outcomes. You do not need advanced probability, only the notion that when a model is given a partial sequence, it assigns a number (between 0 and 1) to each possible next token, and these numbers sum to 1. The larger the number, the more likely that token is. The core operation is nothing more than comparing probabilities. In standard generation, at each step you look at the distribution and pick the token with the highest probability (or sample according to temperature). That is one forward pass per token. 

In multi-token decoding, you let a small model compute a draft: it guesses a sequence of, say, four tokens. Then you ask the large model to look at the sequence that includes those four guesses (plus the context before them) and, in a single pass, compute probabilities for each position. For each position, you check whether the draft token you proposed there is indeed the most likely token according to the large model. If it is, you accept it and move to the next. If it is not, you stop — you have just found the first place where the large model disagrees — and you use the large model's own chosen token at that position. Everything after that is discarded, and you start a new draft from that corrected token. The overall output is exactly what standard generation would have produced, because every accepted token is one the large model would have picked, and the first rejected token is replaced by the large model's pick. 

The number of tokens you can draft and the probability that the draft is right determine the speed-up. If the draft is usually right, you accept several tokens per pass; if it is often wrong, you waste a verification pass and gain nothing. The mathematics of the acceptance rule is the key: you must use the same sampling distribution for the large model as you would in standard generation, so that the probability of accepting a token equals the probability the large model would have generated it. This is exactly the condition that makes the output distribution identical to standard greedy or sampled generation.

Let p_t be read 'p sub t'. It means the probability distribution over the vocabulary at position t, as computed by the large model. The draft proposes a sequence d_1, d_2, ..., d_k. Here d_t is read 'd sub t' and means the token proposed by the draft at position t. For each t, accept d_t if d_t equals the arg max of p_t(v). This means d_t is the token with the highest probability in the distribution p_t. Or, if you are sampling, accept d_t if it is drawn from p_t. Stop at the first t where d_t does not equal the arg max of p_t(v), and replace that position with the large model's own token v_star, which is read 'v sub star' and equals the arg max of p_t(v). The number of tokens generated per forward pass is N. If we stop at position t (meaning positions 1 through t-1 were accepted, but t was rejected), then N equals t minus 1 plus 1, which simplifies to t. Wait, let us be precise. If we accept k tokens and the (k+1)-th token is rejected, we have generated k tokens in that pass. The formula given in the original text was confusing. Let us define it clearly: Let k be the number of consecutive draft tokens that match the large model's top choice. Then the number of new tokens added to the output in this step is k. If all k draft tokens match, we add k tokens. If the first one fails, we add 0 tokens (we discard the draft and replace the first token with the correct one, but that replacement counts as the start of the next logical step or is included in the 'corrected' state). Actually, standard acceptance adds the accepted tokens. So if k tokens are accepted, we gain k tokens. The original text said N = min(t-1, k) + 1. This is incorrect for counting *newly generated* tokens in a single pass. If we verify k tokens and all are correct, we generate k tokens. If the first one is wrong, we generate 0 draft tokens (we just fix the current position). So N is simply the count of accepted draft tokens.


## A worked example

Suppose the vocabulary has only five tokens: A, B, C, D, E. The context (the prompt plus tokens already generated) is fixed. The large model, if asked for the next token, would produce a probability distribution over those five. Standard generation picks the most likely. In this example, we will draft three tokens and verify them.

Step 1: Large model distribution at position 1: A:0.50, B:0.20, C:0.15, D:0.10, E:0.05. These are the probabilities the big model assigns to each possible next token given the context so far. The most likely is A (0.50).

Step 2: Draft model proposes: A, then B, then C. The small model guesses three tokens ahead. It has seen the same context and generated A, then based on A it generated B, then based on 'A B' it generated C.

Step 3: Verification pass: big model scores the sequence 'A B C' in one forward pass. The big model processes the whole sequence at once. It outputs a distribution for each position, given the prefix before that position.

Step 4: Position 1: distribution is A:0.50, B:0.20, ... — most likely is A, which the draft proposed. Accept A.

Step 5: Position 2: given 'A', distribution is B:0.30, A:0.25, ... — most likely is B, which the draft proposed. Accept B.

Step 6: Position 3: given 'A B', distribution is D:0.40, C:0.20, ... — most likely is D, not C. Reject C and everything after. The draft proposed C, but the big model prefers D. So we stop. The token at position 3 becomes D (the big model's pick), and we start a new draft from 'A B D'.

Step 7: Final generated sequence: A, B, D, ... We produced two tokens (A and B) in one verification pass, instead of two separate passes. The next pass will draft starting from 'A B D'.

Result: We generated 2 tokens in 1 forward pass. If the draft had been correct 3 tokens, we would have generated 3 tokens in 1 pass — a 3x speedup. The output A B D is exactly what standard generation would have produced, because each accepted token was the big model's top choice, and the rejected position was replaced by the big model's own choice.


## From scratch, in Python

```python
import random

# We'll simulate a large model's probability distributions for a toy vocabulary.
# In reality, these come from a neural network forward pass.

vocab = ['A', 'B', 'C', 'D', 'E']

# A very simple 'large model': given a context (a string), return a probability distribution.
# We'll hard-code it for this example so we can see the mechanics.

def large_model_probabilities(context):
    # In a real LLM, this would be a huge tensor of probabilities.
    # Here, we just return a fixed distribution, but in a real system it depends on context.
    # We'll make it so that the large model prefers tokens in a certain pattern.
    if context == '':
        return {'A': 0.50, 'B': 0.20, 'C': 0.15, 'D': 0.10, 'E': 0.05}
    elif context == 'A':
        return {'B': 0.30, 'A': 0.25, 'C': 0.20, 'D': 0.15, 'E': 0.10}
    elif context == 'AB':
        return {'D': 0.40, 'C': 0.20, 'B': 0.15, 'A': 0.15, 'E': 0.10}
    else:
        return {'A': 0.20, 'B': 0.20, 'C': 0.20, 'D': 0.20, 'E': 0.20}

# A simple draft model: we'll simulate it as a function that proposes a sequence of tokens.
# In practice, it's another neural network, but smaller and faster.

def draft_model_generate(context, k):
    # For simplicity, we'll just return a fixed draft 'A B C' if context is empty.
    # In a real system, it would autoregressively generate k tokens.
    if context == '':
        return ['A', 'B', 'C']
    else:
        # Otherwise, just return a random guess for illustration
        return random.choices(vocab, k=k)

# Standard autoregressive generation (one token per forward pass)
def standard_generate(context, max_tokens):
    output = context
    for _ in range(max_tokens):
        probs = large_model_probabilities(output)
        # pick the most likely token (greedy)
        next_token = max(probs, key=probs.get)
        if next_token is None:  # in a real system, there is an end-of-sequence token
            break
        output += next_token
    return output

# Speculative decoding with a draft model and verification
def speculative_generate(context, max_tokens, draft_length):
    output = context
    # While we haven't reached the maximum output length
    while len(output) - len(context) < max_tokens:
        # Draft a sequence of tokens
        draft = draft_model_generate(output, draft_length)
        # Verify each position in one forward pass (but we'll simulate it step by step for clarity)
        accepted = 0
        for i in range(draft_length):
            probs = large_model_probabilities(output)
            top_token = max(probs, key=probs.get)
            if draft[i] == top_token:
                output += draft[i]
                accepted += 1
            else:
                # Reject and replace with the large model's choice
                output += top_token
                break
        else:
            # All accepted, we can break out of the while loop if we have produced enough
            if accepted == draft_length:
                break
    return output

# Test it
context = ''
print('Standard:', standard_generate(context, 5))
print('Speculative:', speculative_generate(context, 5, 3))
```


## The same thing, with a framework

```python
In practice, you would not write the verification loop yourself. Libraries like Hugging Face's transformers or vLLM provide speculative decoding out of the box. For example, in vLLM you can set `--speculative-model` and `--num-speculative-tokens`. Under the hood, vLLM runs the draft model to produce k tokens, then runs the large model on the concatenated sequence, and uses the acceptance rule to decide which tokens to keep. The `from_scratch` code shows the exact logic: the draft proposes, the large model scores, and the first disagreement truncates.
```


## What the code does

The code starts by defining a toy vocabulary and a simulated `large_model_probabilities` function. In a real LLM, this function would be the output of a neural network — a tensor of probabilities over the whole vocabulary, given the context. Here, it returns a Python dictionary mapping each token to a probability. The dictionary is a Python data structure that looks up a value (the probability) by a key (the token) — you have seen dictionaries in your Python notes; they let you access items by name rather than by position. Then we define `draft_model_generate`, which in a real system would be a smaller, faster model that autoregressively produces a sequence of k tokens. Here it just returns a fixed list `['A', 'B', 'C']` for an empty context. Then there are two functions. `standard_generate` loops, each time calling `large_model_probabilities` and picking the token with the highest probability using `max(probs, key=probs.get)`. That is the one-token-per-pass baseline. The `for` loop runs `max_tokens` times, each iteration appending one token to the output string. `speculative_generate` is the interesting one. It runs a `while` loop as long as we have not produced the desired number of tokens. At each iteration, it first calls the draft model to get a list of proposed tokens. Then it iterates through that list, checking each one: for each position, it asks the large model for the probability distribution given the current output, finds the most likely token, and compares it to the draft token. If they match, it appends the draft token and moves to the next position. If they differ, it appends the large model's top token and breaks out of the loop, discarding the rest of the draft. After the loop, if all draft tokens were accepted, we break out of the while loop too. The key line is `if draft[i] == top_token` — that is the acceptance rule. The `for ... else` construct is a Python feature: if the loop completes without a `break`, the `else` block runs. Here, it checks if we accepted all draft tokens, and if so, we break the outer while loop because we have finished. Note that the output of `speculative_generate` is exactly the same as `standard_generate` for this toy, because the draft is always a subset of what the large model would pick. In a real system, the draft might be wrong, and the output after the first rejection becomes the large model's choice, still identical to standard generation.


## Common misconceptions

**Speculative decoding changes the output quality** — If the draft model is wrong, the final output is worse than what the big model would have produced.

The acceptance rule ensures that the final output is exactly what the large model would have generated. Every accepted token is one the large model would have picked, and the first rejected token is replaced by the large model's own choice. There is no deviation. In fact: The output distribution is identical to standard generation (greedy or sampled), regardless of the draft model's accuracy. The draft only affects speed, not quality.

**Speculative decoding doubles the compute, so it must be slower** — Running a second model and a verification pass uses more compute, so it must increase latency.

The draft model is much smaller and faster, so its compute is negligible compared to the big model. The verification pass processes several draft tokens in one forward pass, so it replaces multiple sequential forward passes. The tradeoff is compute for latency: you spend a bit more FLOPs but reduce wall-clock time. In fact: The net effect is a latency reduction, not an increase, as long as the draft model is a good predictor. The compute cost of the draft is amortised over several tokens.

**The draft model must be trained separately for each large model** — You need a different small model for every big model, and it has to be trained with special alignment.

Any small model that can predict the same text distribution works, as long as it has seen similar data. Often people use a smaller version of the same model family or even a pruned version of the big model. No special training is required; it just needs to be reasonably good at predicting the big model's choices. In fact: A good draft model is one that has a high acceptance rate, but it can be any fast model, even a different architecture. You can also use the big model itself with a smaller 'draft head' or early exit.

**Speculative decoding is only for greedy decoding** — You can only use it when you always pick the most likely token; with sampling, it would break.

The acceptance rule can be adapted to any sampling scheme. For sampling, you accept a draft token with a probability that is the ratio of the draft model's probability to the large model's probability, and otherwise you reject. This preserves the exact distribution. In fact: Speculative decoding works for both greedy and stochastic sampling, as long as the acceptance rule matches the sampling distribution. Many libraries implement an adjusted rule to guarantee correctness.

**Drafting more tokens always gives a bigger speedup** — If I set the draft length to 10, I get 10 tokens per verification pass, so it must be 10x faster.

The longer the draft, the more likely the draft model will make a mistake, and the verification pass is wasted after the first mistake. There is an optimal draft length, usually 4–6, that balances the cost of a bad draft against the benefit of multiple accepted tokens. In fact: The speedup is limited by the acceptance rate. If the draft is wrong early, you gain nothing. The optimal draft length is found empirically based on the draft model's accuracy.


## In short

Generation is sequential: one token per forward pass, each forward pass cannot start until the previous token is known. Multi-token decoding breaks this by letting a small, fast draft model guess several future tokens, then having the large model verify them all in a single parallel pass. The large model accepts the draft up to the first disagreement, then takes over from there. Because every accepted token is exactly what the large model would have chosen, the output is identical to standard generation, and the speedup comes from reducing the number of sequential passes. In practice, you enable this in your serving framework (vLLM, TensorRT-LLM) by specifying a draft model and draft length; the acceptance rate determines the actual speedup. If the draft is poor, you may even slow down, so it is a tuning problem, not a free lunch. It is a core technique for making LLM applications feel instant, especially in interactive, agentic, and on-device settings.


## Where this is used

In large language models, multi-token decoding is a standard inference optimization used by production systems like vLLM, TensorRT-LLM, and Hugging Face's text-generation-inference. A developer running a server for a large model such as Llama 3 70B would enable speculative decoding by specifying a smaller draft model (e.g., a 1B parameter version of the same family) and the number of speculative tokens (often 4–5). The server then handles the draft-and-verify loop automatically. The benefit is a 2–3x reduction in latency for greedy decoding, without any loss of quality because the output is provably identical. When the draft is wrong, the server wastes a forward pass, but statistically, a good draft model is right often enough to give a net win. Small language models running on devices (phones, laptops, edge devices) have even tighter latency budgets. Here, multi-token decoding is critical because the device's small model is fast, but still sequential — and memory bandwidth, not compute, is the bottleneck. A tiny draft model (sometimes the same model with a reduced precision) can run on the device, while the larger model runs in the cloud; the device drafts, the cloud verifies. This is a common pattern in on-device assistants: draft locally, verify remotely, and only the corrections go over the network, saving bandwidth. In retrieval-augmented generation (RAG), the generation bottleneck is still the LLM. RAG systems spend significant time retrieving, but once the context is assembled, speculative decoding speeds up the answer generation. The draft model has the same context, and the large model verifies the draft. Since RAG is often used in interactive apps, reducing generation latency by 2x directly improves the user experience. In agentic systems that plan and call tools, agents generate chains of thought and tool calls. These can be long, and they are inherently sequential. Multi-token decoding is especially impactful here because a reasoning chain might be 500 tokens; a 2x speedup halves the time the agent spends thinking before it acts. Also, some agentic frameworks use a special 'tool-call token' that terminates generation; speculative decoding must handle the case where the draft proposes a tool call, because the verification must not accept a token that would trigger a tool call if the large model would not have picked it. In everyday applied machine learning — recommendation, vision, forecasting, fraud detection — multi-token decoding as described is specific to autoregressive generation, so it does not apply to a prediction that outputs a single value or a label. However, many modern recommendation systems use LLMs to generate natural-language explanations or conversational responses, and those benefit from the same speedup. For a developer building software with LLMs, the practical reality is: you usually do not implement speculative decoding yourself; you turn it on in your serving framework. You choose a draft model (often a smaller version of the same model, e.g., a 1B model for a 70B model) and set the draft length. What breaks when it is done badly: if the draft model is too small or misaligned, it disagrees with the large model so often that you waste more compute on verification than you save — the acceptance rate drops below ~50% and you get no speedup or even a slowdown. Also, if you use sampling instead of greedy decoding, the acceptance rule must be modified to match the sampling distribution exactly, otherwise the output is no longer identical to the original model's output, which may be unacceptable in applications where consistency matters. Finally, the idea extends beyond two models: you can use multiple drafts and pick the best, or use a single model with a faster 'draft head', but the basic mechanism is always the same — draft in parallel, verify sequentially, and discard disagreements.


---

Read this on the web: https://basavapurushottam.com/learn-ai/multi-token-decoding
