# Latency and Speedups

Reducing perceived and actual response time through caching, batching, streaming and decoding strategy.

**Rests on:** Multi-Token Decoding, Quantization

**Leads to:** Frontend Performance Testing

*LLMs and Generative AI · Inference Optimization*


## Why learn this

You are reading this because you want to build or use AI systems that feel fast. If you press a button in an app and the response takes five seconds, you will blame the app. But most of that time is not the model 'thinking' — it is the way the system moves data around. This concept teaches you how to make a model feel faster without making the model itself smaller or smarter. It unlocks the rest of Inference Optimization: quantization (making the model fit in less memory), distillation (training a tiny student to copy a large teacher), and local deployment (running the model on your own laptop instead of a cloud API). Each of those topics assumes you already understand the tricks in this page: caching, batching, streaming, and decoding strategy. Master this and you will never again accept a slow AI system as inevitable — you will know exactly which lever to pull.


## The intuition

Imagine you run a small tea stall near a railway station. The stall has one kettle, and the kettle takes exactly two minutes to boil. You have a queue of customers asking for chai. Each customer asks for the same type of chai. You could boil the kettle fresh for each customer — two minutes each. But that is slow. So you boil a full kettle once, and every customer who asks for the same chai gets a cup instantly. That is caching: you remembered the result and reuse it instead of recomputing. But what if every customer asks for a different chai — some with ginger, some with less sugar? The kettle still takes two minutes, but you can put water for several cups in the same kettle at the same time. Instead of twenty minutes for twenty customers, you boil one large kettle in four minutes and pour twenty cups. That is batching: you combined several jobs into one, taking longer per job but far less time per job. Now, what about the customer who is standing there, waiting? They do not care that the total order will be ready in four minutes; they want something to drink now. You give them a half-filled cup as soon as the water is hot, then refill as soon as the next bit is ready. They start sipping in thirty seconds, even though the full cup takes four minutes. That is streaming: you send the output as it is produced, not all at once. Finally, the 'decoding strategy' is the order you pour the tea. If a customer says 'three chai, two with ginger, one without,' you do not need to boil the kettle and wait for the third chai before you start pouring the first. You start pouring the first one immediately — the model can start generating the first few words while the rest of the sentence is still being decided. All four tricks — caching, batching, streaming, decoding — attack the same enemy: time spent waiting.


## An analogy

Think of a large language model as a librarian who writes a book one page at a time. You ask the librarian: 'Write a one-page summary of the causes of the 1857 uprising.' The librarian must read a huge reference shelf to answer. Without caching, the librarian would re-read the entire shelf for every page they write. With caching, they read the shelf once and keep their notes handy — that is how the model remembers earlier parts of the conversation so it does not have to re-read everything from scratch. The typing speed (the model's generation) is fixed, but the flurry of re-reading is eliminated, so the first page appears almost instantly. Now, suppose twenty readers all ask slightly different summaries on the same shelf. The librarian could read separately for each — twenty times the shelf reading. Or they could gather all the readers, read the shelf once, and then write all twenty summaries in sequence — the total time might be longer for the first reader, but everyone gets their answer sooner. That is batching. But here is the catch: if the librarian types the entire summary for the first reader before starting the second, the first reader sees a long pause — the page with the title appears, then nothing for a minute. Instead, the librarian types the first line of summary A, then the first line of summary B, then the second line of A, and so on. Everyone sees a line appear every few seconds, even though the last line arrives at the same time as it would have with no batching. Perceived latency drops while throughput stays high. Finally, the decoding strategy: the librarian does not have to decide the final word of a sentence before writing the first word. They can start 'In 1857…' and decide the rest while the reader is already reading. With greedy decoding, they choose the most likely next word and move on. With beam search, they keep several possible continuations in mind, but that costs time. The choice of decoding strategy is a trade-off: how much time you spend trying to be clever versus just getting the next word out fast.


## Definition

Latency is the time from sending a request to receiving the complete response; speedups are techniques that reduce that time — either by making the arithmetic faster, or by making the response appear faster than it really is (perceived latency), through caching, batching, streaming, and clever decoding choices.


## How it connects

You have not learned any other topic in this library yet — this is your first page. But your library records a set of topics that this one builds toward: Edge and Air-Gapped Inference, Knowledge Distillation, Local Model Deployment, Quantization, and Test-Time Compute. Each of those assumes you understand the latency techniques in this page. Quantization, for instance, makes a model smaller so it fits on a phone — but the phone still needs streaming to feel responsive. Distillation creates a small student model — but a small model's speedup is useless if the serving framework does not batch requests. So think of this page as the foundation of the entire Inference Optimization folder.


## The mathematics

This concept is not a calculation as much as a timing study. It uses arithmetic — nothing beyond comparing counts and times — but the mental model is from a branch of mathematics called queuing theory, though you won't need its formulas. Queuing theory is the study of waiting lines: how long does a customer wait, how busy is the server, how many customers are waiting. The key insight is that a system can be fast for one person but slow for many, or fast for many but slow for one. The arithmetic you'll actually do is just multiplication and addition: count tokens, multiply by time per token, add overheads. And there is one operation that appears everywhere in this concept — the arithmetic average, though you already do it naturally when you say 'it takes about 0.1 seconds per token.' The numbers are small: a model generates 20 tokens per second, so each token takes 50 milliseconds (since 1000 ms = 1 s, 1000 / 20 = 50). Adding a caching trick might cut the first token time from 200 ms to 30 ms — you can subtract and see the difference. No equations beyond that are required, but we will still express it in symbols so you can compare numbers at a glance.

Total response time = T_first token + (N_output - 1) × T_per token
where:
• T_first token = prompt_length × T_process per token
• T_per token = T_generate per token (with KV cache; without, it would be proportional to total context length)

Perceived latency = Time until the first token appears (the rest can stream).


## A worked example

```
You have a chat model that answers one request. The request is 50 words long (about 400 tokens, since 1 token ≈ 4 characters). The model can generate 20 tokens per second. To compute the first token, the model must process the entire prompt once. Processing the prompt takes 100 milliseconds per token of prompt length. After the first token, each token requires processing only the new token (with a KV cache) — 50 milliseconds per token. Without a cache, each new token would need to re-process the whole conversation so far — which grows with each token.

Step 1: 100 ms/token × 400 tokens = 40,000 ms = 40 s   # Time to process the prompt to generate the first token, without any caching.
Step 2: 50 words × 30 tokens/word = 1,500 tokens total output   # Assume the model writes 50 words, each averaging 30 tokens (this is a simplification; actual words per token varies).
Step 3: 1,500 tokens × 50 ms/token = 75,000 ms = 75 s   # Time to generate all remaining tokens after the first, with a KV cache.
Step 4: Total with cache: 40 s (first token) + 75 s = 115 s   # Without KV cache, not only the first token but every token would need to reprocess the prompt.
Step 5: Without cache: each token needs to process (400 + token_number) tokens; summing from token 1 to 1500 gives roughly 1,500 × 400 + (1500²)/2 ≈ 600,000 + 1,125,000 = 1,725,000 ms = 1,725 s   # Rough estimate — re-reading the entire history each time.
Step 6: Speedup = 1,725 s / 115 s ≈ 15 times   # KV caching makes generation about 15× faster in this scenario.

Result: With a KV cache, the total time drops from about 29 minutes to under 2 minutes — a 15× speedup. And the first token arrives in 40 seconds instead of after the entire generation, so perceived latency is much better because you see text start immediately.
```


## From scratch, in Python

```python
# Simulating the speedup from a KV cache

# A very small model: just a function that produces a token
# In a real model, this would be a neural network, but here we
# just return a fixed token after a delay.

def generate_token_without_cache(history_tokens):
    # Pretend each token in the history needs 0.1 ms to process
    time_per_token_ms = 0.1
    # Multiply by the length of history
    return len(history_tokens) * time_per_token_ms

def generate_token_with_cache(cache_length):
    # With a cache, we only process the last token (the new one)
    # so the cost is constant regardless of history length
    return 1 * 0.1  # one token's worth of computation

# Simulate a prompt of 100 tokens, output of 5 tokens
prompt_length = 100
output_length = 5

# Without KV cache
history = list(range(prompt_length))  # placeholder tokens
start_time = 0
for i in range(output_length):
    # Each token adds its own processing plus all previous tokens
    cost = generate_token_without_cache(history)
    start_time += cost
    history.append(i)  # add the new token to history

print(f"Time without cache: {start_time} ms")

# With KV cache
history = list(range(prompt_length))
start_time = 0
# First token still needs to process the full prompt
start_time += generate_token_without_cache(history)
for i in range(output_length - 1):
    # Subsequent tokens only process the newest token
    start_time += generate_token_with_cache(len(history))
    history.append(i)

print(f"Time with cache: {start_time} ms")
```


## The same thing, with a framework

```python
In practice, frameworks like Hugging Face's `transformers` handle the KV cache automatically. When you call `model.generate(...)`, you can pass `use_cache=True` (default) and the library manages a cache object. For streaming, you use `TextIteratorStreamer` to get tokens one by one. Batching is done by `DataLoader` and `pipeline` with `batch_size > 1`.
```


## What the code does

The code defines two functions simulating the time per token. Without a cache, each new token requires re-processing the entire history — hence the loop. With a cache, after the first token, each subsequent token costs a constant amount. The print statements show the dramatic difference. In a real transformer, the hidden states (keys and values) are stored in a tuple and reused — the cost of attending to a past token drops from O(context length) to O(1) for that token.


## Common misconceptions

**Streaming makes the model actually faster** — If the app streams tokens, the model is generating faster.

Streaming only changes the delivery schedule — the model still takes the same total time to generate all tokens. It reduces perceived latency (the user sees something happen sooner) but not the actual total time. In fact: Streaming is a delivery trick. The total time to generate the full response is unchanged, but the time to the first visible token is much less, which makes the app feel faster.

**Batching always speeds up each request** — If I batch requests, every request will finish faster.

Batching increases throughput (requests per second) but can increase the latency of each individual request because requests share compute and may wait for the batch to fill. In fact: Batching raises the number of requests served per second at the cost of higher per-request latency on average. Use it for non-interactive workloads; for interactive ones, use smaller batches or streaming.

**KV cache stores the entire conversation as text** — The KV cache saves a copy of the prompt text.

The KV cache stores the key and value vectors from the attention layers — high-dimensional floating-point numbers — not the original text token IDs. It is a matrix of numbers, not a string. In fact: The KV cache is a set of tensors (lists of lists) representing the attention keys and values. It is derived from the tokens but is not the tokens themselves.

**Smaller models are always faster** — If I use a smaller model, my latency problem is solved.

A smaller model might have lower per-token cost, but the system overhead (framework, tokenization, I/O) can dominate. Also, a small model often produces worse quality, and if the latency problem is due to network round trips, the model size doesn't matter. In fact: Model size is one factor. Before shrinking the model, measure where time goes: prompt processing, generation, network, or tokenization. Often caching or streaming yields bigger wins with no quality loss.

**Streaming and batching are mutually exclusive** — You can either batch or stream, not both.

You can batch multiple requests and stream the output of each request as tokens are generated. The server creates a stream per request and interleaves their token generation. In fact: Batching and streaming combine: multiple requests are processed together, but each request's tokens are sent to its user as soon as they are produced, so each user sees a growing response.


## In short

Latency is the time from request to full response. To speed it up: (1) enable KV caching to avoid re-processing the prompt for each token; (2) batch requests to raise throughput (but accept higher per-request latency); (3) stream tokens so the first appears immediately and the user feels the speed; (4) choose a fast decoding strategy (greedy or top-k) over beam search unless quality demands it.


## Where this is used

**Large Language Models (LLMs):** The dominant cost in serving an LLM is the sequential generation of tokens. Two things matter: the time to first token (TTFT) and the time per output token (TPOT). Caching the key-value pairs from the attention layer (KV cache) turns the per-token cost from 'recompute the entire prompt' to 'just process the new token,' which is often a 10–100× speedup, exactly as your manual calculation showed. Without this, serving an LLM would be impractical — even a 1,000-token prompt would take 100 seconds just to start replying. Every production framework (vLLM, TensorRT-LLM, Hugging Face `transformers`) implements KV caching internally. Developers also use **prefix caching** (store the processed prompt embeddings) so that if several users ask similar questions, the shared prefix is not recomputed. Beyond caching, **batching** is used to increase throughput: you feed multiple user prompts into the GPU at once, amortizing the memory reads of the model's weights across several requests. A single request may see its latency go up slightly (because it's competing for compute), but the server can serve far more requests per second, which reduces queue times. **Streaming** is the user-facing trick: the server sends the first token as soon as it's available, then sends each subsequent token as it's generated. Modern chat interfaces do this by default — you see text appear word by word, which feels fast even if the total generation takes the same time. **Decoding strategy** also matters: greedy decoding (picking the most likely next token) is fastest; beam search (keeping several candidates) can be 2–5× slower because it computes multiple sequences in parallel. For latency-sensitive applications, developers often choose greedy or top-k sampling with a small k. Speculative decoding is a newer technique: a small, fast model drafts several tokens, and the large model verifies them in one forward pass, resulting in a big speedup (2–3×) without changing the output quality. **Small Language Models on devices:** On a phone or laptop, the model must fit in memory, but the same four techniques apply. KV cache is essential because device memory is limited — caching keys and values for the entire conversation can be thousands of elements, but it's far cheaper than recomputing. Batching is often not possible (single user), so streaming is the main lever; a phone app shows tokens appearing immediately, making the model feel quick even if it generates at only 10 tokens per second. **Retrieval-Augmented Generation (RAG):** In RAG, the model answers a question with the help of retrieved documents. The dominant latency often comes from the retrieval step (searching a vector database) and the long prompt (the documents are prepended to the user query). Caching the embeddings of frequently asked questions or the documents' vectors lets you skip re-embedding. Prefix caching of the document section means the model doesn't need to re-process the same documents for each new user query in the same session. Streaming applies here too: the model should start answering as soon as the first token of the answer is ready, not wait for the entire response. Developers using RAG frameworks (LangChain, LlamaIndex) often set `streaming=True` and use cache middleware. **Agentic systems:** Agents that plan, call tools, and act in loops are inherently multi-step: the model decides to call a tool, the tool runs (e.g., a web search), the result is fed back, and the model continues. The latency breakdown is dominated by the number of round trips. Caching the model's reasoning about a tool call (if the same tool is called with the same arguments) can collapse multiple steps. Streaming is critical in agents: the user wants to see the agent's chain of thought as it happens. Batching doesn't apply directly (each agent step is sequential), but **speculative decoding** (using a fast draft model) and **early-exit decoding** (skipping some layers when confidence is high) are active research areas. Frameworks like LangChain support streaming by default. **Everyday applied ML:** Beyond LLMs, latency optimization is ubiquitous in computer vision (object detection on a camera feed batches frames), forecasting models (batching time-series predictions), fraud detection (a scoring model must return within milliseconds — often using a simple model with cached features), and speech recognition (streaming is essential — you can't wait for the user to stop speaking before showing partial transcription). For a developer, the concrete actions are: profile where time goes (first token vs. per token), enable KV caching (it's often a flag `use_cache=True`), use batch inference for non-interactive jobs, and implement streaming for interactive jobs. The most common mistake is using a framework's default settings without checking — for example, not enabling `use_cache` in a custom inference loop, resulting in a 10× slowdown.


---

Read this on the web: https://basavapurushottam.com/learn-ai/latency-and-speedups
