# Quantization

Representing weights and activations at lower numerical precision to cut memory and often latency, with modest and measurable quality loss.

**Rests on:** Data Compression, Forward Propagation

**Leads to:** Latency and Speedups

*LLMs and Generative AI · Inference Optimization*


## Why learn this

Quantization is the key that lets a model that needs gigabytes of memory run on your laptop or phone. Without it, large language models are locked to expensive cloud servers. This unlocks: Local Model Deployment (run models offline and privately), Edge and Air-Gapped Inference (serve on devices with no internet), and makes all of Inference Optimization cheaper. Quantization is also the reason you can download a 7-billion-parameter model with Ollama and get responses in seconds on a consumer laptop.


## The intuition

A computer stores numbers in chunks called bits. A 32-bit number can represent billions of different values. But do we always need that much precision? Think of weighing rice at a shop. The shopkeeper can use a digital scale that shows grams to three decimal places, or a rough scale that shows to the nearest 10 grams. For most purposes, the rough scale is fine and much cheaper. Similarly, a model's weights are just numbers that tell it how strongly to listen to each input. If we round those numbers from 32-bit precision down to 8-bit, we still capture the essential spread, just with less detail. The model's output changes slightly—like the rice being a few grams off—but the meaning stays the same. Quantization is exactly this rounding, done at scale, across millions of numbers, to save memory and speed things up.


## An analogy

Imagine you are giving directions to a friend over a phone that has a very weak signal. You can either send every detail—every tiny turn, every bump in the road—and have it take a long time and drain battery, or simplify. You decide to round distances: instead of "walk 1.03 km then turn right after 0.47 km", you say "walk 1 km, turn right, walk half a km". The friend still gets there, maybe a few meters off, but the message is shorter and faster. In the model, the weights are the directions. Quantization rounds them to fewer digits. The directions still work, but if you round too much—say, to the nearest kilometer—your friend might walk into a wall. This is the limit: round too coarsely and the model's answers become nonsense. But the sweet spot—8-bit, or even 4-bit—keeps most of the quality while cutting memory by 75% or more. The analogy breaks down because a model has billions of weights, so even small rounding errors can accumulate. But the principle is the same: we trade a little accuracy for a lot of savings.


## Definition

Quantization is representing each weight and activation in a model using fewer bits than the usual 32-bit floating point, reducing the model's memory footprint and often speeding up computation, at the cost of a small, measurable degradation in output quality.


## How it connects

You are learning Inference Optimization. Quantization is one of the main tools in that toolbox, alongside Knowledge Distillation (which shrinks the model by training a smaller copy) and Local Model Deployment (where quantized models are loaded into Ollama and run on a laptop). All of these make serving models cheaper and faster. Quantization specifically is why you can run a 7B model on 8GB RAM: the file is roughly a quarter the size.


## The mathematics

Quantization is a way to store numbers using less space by replacing precise values with simpler integers. Imagine you have a list of weights from a neural network. These weights are real numbers, like 0.25 or -1.3. To save memory, we want to represent each weight as a small integer, such as a number between 0 and 255 (which fits in 8 bits).

The process works in two phases: quantization (compressing) and dequantization (expanding).

Phase 1: Quantization (Compressing)
First, we look at all the weights in the model to find the smallest value and the largest value. Let us call the smallest value 'min' and the largest value 'max'.

We define a 'step size'. This is the width of each small interval or bin that covers the range from min to max. The number of possible integer values depends on how many bits we use. If we use 'bits' bits, the number of distinct integers we can store is 2 raised to the power of 'bits'. For example, with 8 bits, we have 2^8 = 256 possible integers (from 0 to 255). With 4 bits, we have 2^4 = 16 possible integers (from 0 to 15).

The step size is calculated by taking the total range (max minus min) and dividing it by one less than the number of possible integers. We subtract one because the integers start at 0. So:
step_size = (max - min) divided by (2 raised to the power of bits, minus 1).

Now, for any single weight 'w' in our list, we find which bin it falls into. We do this by shifting the value so that 'min' becomes zero, then dividing by the step size.
The formula is:
integer_value = round( (w - min) divided by step_size )

Here, 'round' means finding the nearest whole number. If the result is 0.4, it rounds to 0. If it is 0.6, it rounds to 1. The integer_value will be an integer between 0 and (2^bits - 1).

Phase 2: Dequantization (Expanding)
Later, when we need to use the weight, we convert the integer back to a real number. This is called dequantization. We take the stored integer, multiply it by the step size, and then add 'min' back.
The formula is:
dequantized_w = integer_value * step_size + min

This gives us an approximation of the original weight. The error comes from the fact that we rounded to the nearest integer, so the dequantized value might be slightly different from the original 'w'.

Symbol glossary:
- w: a single weight value (a real number)
- min: the smallest weight in the entire set
- max: the largest weight in the entire set
- bits: the number of bits used to store each integer (e.g., 8 or 4)
- step_size: the width of each bin, calculated as (max - min) / (2^bits - 1)
- round(): the operation of finding the nearest whole number
- integer_value: the compressed integer stored in memory
- dequantized_w: the approximated real number recovered from the integer


## A worked example

We have three weights: 0.25, -0.50, and 1.00. We decide to quantize these using 8 bits. This means we can store integers from 0 to 255 (since 2^8 = 256).

Step 1: Find the range.
The smallest weight is min = -0.50.
The largest weight is max = 1.00.
The total range is max minus min:
range = 1.00 - (-0.50) = 1.00 + 0.50 = 1.50.

Step 2: Calculate the step size.
With 8 bits, the number of intervals is 2^8 - 1 = 256 - 1 = 255.
step_size = range / 255
step_size = 1.50 / 255
step_size ≈ 0.00588235...
(We will use the approximate value 0.00588 for clarity, but keep more precision in intermediate steps if needed.)

Step 3: Quantize the first weight, w = 0.25.
First, shift by min:
w - min = 0.25 - (-0.50) = 0.25 + 0.50 = 0.75.
Next, divide by step_size:
scaled_value = 0.75 / 0.00588235...
scaled_value ≈ 127.5
Finally, round to the nearest integer:
integer_value_1 = round(127.5) = 128.
So, 0.25 is stored as the integer 128.

Step 4: Quantize the second weight, w = -0.50.
Shift by min:
w - min = -0.50 - (-0.50) = 0.
Divide by step_size:
scaled_value = 0 / 0.00588235... = 0.
Round to nearest integer:
integer_value_2 = round(0) = 0.
So, -0.50 is stored as the integer 0.

Step 5: Quantize the third weight, w = 1.00.
Shift by min:
w - min = 1.00 - (-0.50) = 1.50.
Divide by step_size:
scaled_value = 1.50 / 0.00588235... ≈ 255.0
Round to nearest integer:
integer_value_3 = round(255.0) = 255.
So, 1.00 is stored as the integer 255.

Result: The stored integers are 128, 0, and 255.

Step 6: Dequantize to check accuracy.
To get back the real numbers, we use the formula: dequantized_w = integer_value * step_size + min.

For the first weight (integer 128):
dequantized_w_1 = 128 * 0.00588235... + (-0.50)
dequantized_w_1 ≈ 0.75294... - 0.50
dequantized_w_1 ≈ 0.25294...
This is very close to the original 0.25. The small difference is the quantization error.

For the second weight (integer 0):
dequantized_w_2 = 0 * 0.00588235... + (-0.50)
dequantized_w_2 = -0.50
This matches the original exactly because -0.50 was the minimum value.

For the third weight (integer 255):
dequantized_w_3 = 255 * 0.00588235... + (-0.50)
dequantized_w_3 ≈ 1.50 - 0.50
dequantized_w_3 = 1.00
This matches the original exactly because 1.00 was the maximum value.

In this specific example, the min and max values are preserved exactly. Other values, like 0.25, have a small error (about 0.003) due to rounding to the nearest bin.


## From scratch, in Python

```python
# We'll quantize a list of weights from float to int8.
# A list in Python is an ordered collection.

weights = [0.25, -0.50, 1.00, 0.37, -0.12]

# Find the min and max, using built-in min and max functions.
w_min = min(weights)
w_max = max(weights)

# We choose int8: 256 levels (0 to 255).
bins = 256
step = (w_max - w_min) / (bins - 1)

# Loop over each weight and compute the quantized integer.
quantized = []
for w in weights:
    # Scale: shift so w_min becomes 0, then divide by step.
    scaled = (w - w_min) / step
    # Round to nearest integer (round function).
    q = round(scaled)
    # Clamp to [0, 255] just in case.
    if q < 0:
        q = 0
    if q > bins - 1:
        q = bins - 1
    quantized.append(q)

# Decode back to approximate floats.
dequantized = []
for q in quantized:
    w_approx = q * step + w_min
    dequantized.append(w_approx)

print("Original:", weights)
print("Quantized:", quantized)
print("Dequantized:", dequantized)

# Original: [0.25, -0.5, 1.0, 0.37, -0.12]
# Quantized: [128, 0, 255, 145, 90]
# Dequantized: [0.250, -0.500, 1.000, 0.374, -0.124]
```


## The same thing, with a framework

```python
In PyTorch, you would use torch.quantization.quantize_dynamic to quantize a model's linear layers. Under the hood, it does the same scaling and rounding we did by hand, but across all weights at once. For example:

import torch
model = torch.nn.Linear(4, 4)  # a layer with weights
quantized_model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

This converts the linear layer's weights from float32 to int8 and stores the scale and zero point. It is exactly the same math: find min and max, compute scale, round.
```


## What the code does

First we create a list of weights. A list is a Python container that holds items in order. We use the built-in min and max functions to find the range. We set bins = 256 for 8-bit. The step size is the range divided by 255. Then we loop over each weight using a for loop: we subtract w_min from w to shift the range to start at 0, divide by the step to get a number between 0 and 255, round it to the nearest integer, and clamp to the valid range. We store these integers in a new list. To decode, we loop again, multiplying each integer by step and adding w_min, which gives back an approximate float. The print statements show the original and the dequantized values are nearly identical—the maximum error here is about 0.004.


## Common misconceptions

**Quantization makes the model dumb** — If I round the weights, the model will start gibberish.

The weights are many, but each is only slightly off. The errors tend to cancel out across layers, so the overall output changes only a little—a 1-2 BLEU point drop or a few points of accuracy, not gibberish. In fact: Quantization introduces small errors, but the model is surprisingly robust. For 8-bit, quality loss is often negligible; for 4-bit, it's noticeable but still usable. Extreme quantization like 2-bit can hurt badly, but that's a deliberate trade-off.

**Quantization is the same as pruning** — Quantization and pruning are the same trick—both reduce model size.

Pruning removes weights entirely (sets them to zero). Quantization keeps all weights but stores them in fewer bits. In fact: Pruning reduces the number of weights; quantization reduces the precision of each weight. They can be combined, but they are different techniques.

**Lower bits always speed up inference** — If I use 4-bit, it will always be twice as fast as 8-bit.

Speedups depend on hardware support. Some CPUs or GPUs don't accelerate int8 or int4 operations, so you see memory savings but not full speed gains. Also, the overhead of dequantization can offset the gains for small models. In fact: Quantization reliably cuts memory usage, but latency gains are hardware-dependent. Modern GPUs (like NVIDIA's tensor cores) love int8 and int4, but an old CPU may not. Always benchmark.

**Quantization loss is permanent** — Once I quantize, the model is ruined forever.

Quantization is reversible if you keep the original float32 model. You can always go back or re-quantize with different settings. In fact: Quantization is a deployment-time transformation. You keep the original model, and you can quantize it as many times as you like, with different bit widths, until you find the best trade-off.

**Quantization-aware training is the same as post-training quantization** — I can just quantize my model after training and it's the same as training it quantized.

Post-training quantization is the cheapest and most common way, but it can lose more quality. Quantization-aware training (QAT) simulates quantization during training, so the model learns to be robust to the rounding errors, preserving quality. In fact: If you plan to quantize, QAT is the better path: it bakes quantization tolerance into the weights. Post-training quantization is a quick fix, but for aggressive 4-bit quantization QAT is recommended.


## In short

Quantization stores weights in fewer bits by mapping each number to the nearest integer bin, cutting memory by 75% for 8-bit and 87.5% for 4-bit, with a small quality cost. Use it to make models fit on local devices. Prefer 8-bit for quality-critical tasks, 4-bit for portability, and consider quantization-aware training if the loss is too high.


## Where this is used

Quantization is the reason local deployment works. Without it, a 7-billion-parameter model in 32-bit floats would need 28 GB of memory—more than most laptops have. With 4-bit quantization, the same model fits in about 3.5 GB, which is why Ollama can run it. For edge and air-gapped inference, models are quantized during export to run on phones, Raspberry Pis, or isolated servers with no internet. In retrieval-augmented generation, the embedding model and sometimes the LLM are quantized to fit on smaller hardware—embedding models are often 8-bit to reduce memory when indexing millions of documents. Agentic systems that call tools and plan use quantized LLMs because the agent runs on local hardware alongside the planning loop; a quantized model is fast enough to iterate. In everyday applied ML, developers convert trained models to 8-bit before putting them in production APIs, using tools like ONNX Runtime, TensorRT, or PyTorch's quantization API. What they write is typically a single quantize call that scans the model's weights, computes the scale, and replaces the weight tensors. When done badly—like quantizing without watching the loss—the output quality drops sharply, as measured by perplexity or BLEU. The Craft of Post-Training includes a whole chapter on quantization because it is the most common way to bridge the gap between a training-heavy model and a production constraint.


---

Read this on the web: https://basavapurushottam.com/learn-ai/quantization
