# LoRA and QLoRA

Training small low-rank matrices alongside frozen base weights, with QLoRA additionally quantising the base to fit larger models on modest hardware.

**Rests on:** Foundation Models, Singular Value Decomposition

**Leads to:** QLoRA for Finance

*LLMs and Generative AI · Fine-Tuning*


## Why learn this

This lets you adapt a large, general-purpose AI model to your specific needs using only the computer hardware you already own, without needing a massive server farm or cloud subscription. You will learn how to customize a model for your company’s unique vocabulary, serve multiple different users from one base system, and understand the practical mechanics behind the modern AI techniques that are becoming standard in professional workflows.


## The intuition

Think of a massive pretrained model as a giant library of general knowledge — it has read almost everything on the internet. But if you want it to speak your company's legal jargon, or understand your customer support tickets, you need to 'teach' it a new dialect. In traditional fine-tuning, you would update every single one of its billions of weights — like rewriting every page in the library. With LoRA, you don't touch the library at all; instead, you write a small, dense booklet that sits alongside it, capturing just the new dialect. When you need an answer, you consult the library first, and let the booklet steer the response. QLoRA takes this further by compressing the library into a more compact format, so you can carry it in your backpack (or on your laptop's RAM) instead of needing a warehouse.


## An analogy

Imagine you are a musician who has spent years mastering a grand piano. The piano has thousands of strings, and each string is a weight. To play a new genre — say, jazz — you don't need to restring the entire piano; you only add a few small 'pedals' that subtly bend the notes in a jazzy way. Those pedals are your LoRA adapters: tiny, specialised adjustments. You can have multiple sets of pedals for different genres, and you can swap them in and out on the same piano. QLoRA is like taking a smaller practice piano that has most of the strings — but each string is thinner, cheaper, and just slightly less accurate. Yet with the pedals, the sound is almost identical to the full concert grand. Where the analogy breaks down: the 'pedals' aren't physically attached to one piano; they are mathematical adjustments that work in parallel with the frozen weights — you compute a tiny correction and add it to the frozen output. Also, the piano doesn't 'learn' the pedals — you, the engineer, train them on data, but the piano's base strings never change.


## Definition

LoRA (Low-Rank Adaptation) is a method for fine-tuning that trains a small number of new weights (called adapters) that are added to the frozen original weights of a model, so only the small adapters change during training. QLoRA (Quantised LoRA) additionally compresses the frozen original weights into a smaller numerical format, reducing memory use enough to fit huge models on a single consumer GPU.


## How it connects

You already know from your notes on 'Parameter-Efficient Fine-Tuning' that the default is to adapt a pretrained model rather than train from scratch. LoRA is the most popular parameter-efficient fine-tuning method. Your notes on 'Multi-Tenant Adapter Serving' and 'Quantisation' connect directly: with LoRA, one frozen base can serve many tenants by loading their individual adapters on demand — exactly the economics your notes describe. QLoRA adds the quantisation idea from your notes, which makes on-premises tuning feasible under data residency rules, as in 'QLoRA for Finance'. Under the same parent concept, this also connects to 'Catastrophic Forgetting' — because frozen weights stay untouched, LoRA reduces the risk of losing general ability — and to 'Chat Format Templates', because your tuning data must be shaped into that role-tagged structure before any of this works.


## The mathematics

To understand LoRA (Low-Rank Adaptation) and QLoRA, we need to look at how a neural network makes decisions using tables of numbers called matrices.

Imagine a large table of numbers, which we will call the Weight Matrix. Let us give it the name W. In a large language model, this table might have thousands of rows and thousands of columns. When the model processes information, it multiplies an input vector (a list of numbers representing the current state) by this matrix W to produce an output.

In standard fine-tuning, you would change every single number inside W to make the model better at a new task. This is expensive because there are billions of numbers.

LoRA takes a different approach. It keeps the original table W frozen (unchanged). Instead, it adds a small correction to the result. The key idea is that this correction does not need to be a full-sized table. It can be built from two much smaller tables multiplied together.

Let us define the size of the original matrix W. Suppose W has C rows and D columns. We call C the 'output dimension' and D the 'input dimension'.

LoRA introduces two new small matrices:
1. Matrix A, which has R rows and D columns. We call R the 'rank'. The rank is a small number, much smaller than C or D (for example, if C and D are 4096, R might be just 8).
2. Matrix B, which has C rows and R columns.

Because A has D columns and B has C rows, we can multiply them. But wait, the dimensions must match for multiplication. The standard LoRA formulation adds the correction to the weight matrix before it multiplies the input. So the effective weight becomes (W + BA^T). Let us break down the shapes:
- A is R by D.
- B is C by R.
- To multiply B and A, we need the columns of B to match the rows of A. They do not. However, in the formula X(W + BA^T), we look at the term BA^T.
- A^T (A transpose) flips A. If A is R by D, then A^T is D by R.
- Now we multiply B (C by R) by A^T (D by R)? No, that does not work either. Let us look at the standard definition carefully.

The correct multiplication order for the correction term added to W is B times A^T? Or A times B^T? Let us check the shapes required to match W (C by D).
- We need a result of C rows and D columns.
- If we take B (C by R) and multiply it by A^T (D by R)? No, inner dimensions R and D do not match.
- If we take B (C by R) and multiply it by A (R by D)? Yes. The inner dimensions are both R. The result is C by D. This matches the shape of W.

So the correction matrix Delta_W is calculated as:
Delta_W = B multiplied by A.
Here, B is a table of size C rows by R columns. A is a table of size R rows by D columns. When you multiply them, you get a table of size C rows by D columns.

Why is this useful? The number of values in the original matrix W is C times D. The number of trainable values in LoRA is the values in B plus the values in A. That is (C times R) + (R times D). Since R is very small, this total is much smaller than C times D.

For example, if C = 4096 and D = 4096:
- Original weights: 4096 * 4096 = 16,777,216 numbers.
- With R = 8:
  - B has 4096 * 8 = 32,768 numbers.
  - A has 8 * 4096 = 32,768 numbers.
  - Total trainable numbers: 65,536. This is a tiny fraction of the original.

The formula for the output of this layer becomes:
Output = Input multiplied by (W + Delta_W).
Or more explicitly:
Output = Input multiplied by W + Input multiplied by (B multiplied by A).

There is one more detail. The matrices B and A are initialized in a specific way to ensure the model starts unchanged. Matrix A is filled with random small numbers. Matrix B is filled entirely with zeros. Because B is zero, the product B times A is zero at the start. So the correction Delta_W is zero. The model behaves exactly like the original frozen model until training begins.

QLoRA adds one more step. It takes the frozen matrix W and compresses it into a lower-precision format (like 4-bit integers instead of 16-bit floats) to save memory. However, during the calculation of the correction B times A, it uses full precision. This allows you to run large models on smaller hardware without losing the benefits of LoRA.

The scaling factor alpha (α) mentioned in some texts is used to control how large the correction is relative to the original weights. The effective correction is often written as (alpha / r) times (B times A). This ensures that even if we change the rank R, the magnitude of the update remains stable.


## A worked example

Let us walk through a small example with real numbers.

Suppose our original Weight Matrix W has 2 rows and 3 columns. So C = 2 and D = 3.
W = [[1, 2, 3],
     [4, 5, 6]]

We choose a rank R = 1. This means our small matrices will have only 1 column (for A) or 1 row (for B).

Step 1: Define the shapes.
- Matrix B must have C rows and R columns. So B is 2 rows by 1 column.
- Matrix A must have R rows and D columns. So A is 1 row by 3 columns.

Step 2: Initialize the matrices.
In LoRA, we start with B as all zeros. Let us say we have trained for a while, and now B has learned some values. Let:
B = [[0.1],
     [0.2]]
And A has learned some values. Let:
A = [[0.5, 1.0, 1.5]]

Step 3: Calculate the correction matrix Delta_W.
Delta_W is the product of B and A.
We multiply a (2x1) matrix by a (1x3) matrix.
The result will be a (2x3) matrix.

The element in the first row, first column of Delta_W is:
(First element of B's first row) * (First element of A's first row)
= 0.1 * 0.5 = 0.05

The element in the first row, second column:
(First element of B's first row) * (Second element of A's first row)
= 0.1 * 1.0 = 0.10

The element in the first row, third column:
(First element of B's first row) * (Third element of A's first row)
= 0.1 * 1.5 = 0.15

The element in the second row, first column:
(Second element of B's first column) * (First element of A's first row)
= 0.2 * 0.5 = 0.10

The element in the second row, second column:
(Second element of B's first column) * (Second element of A's first row)
= 0.2 * 1.0 = 0.20

The element in the second row, third column:
(Second element of B's first column) * (Third element of A's first row)
= 0.2 * 1.5 = 0.30

So Delta_W = [[0.05, 0.10, 0.15],
               [0.10, 0.20, 0.30]]

Step 4: Add the correction to the frozen weights.
W_new = W + Delta_W

First row of W_new:
[1 + 0.05, 2 + 0.10, 3 + 0.15] = [1.05, 2.10, 3.15]

Second row of W_new:
[4 + 0.10, 5 + 0.20, 6 + 0.30] = [4.10, 5.20, 6.30]

W_new = [[1.05, 2.10, 3.15],
         [4.10, 5.20, 6.30]]

Step 5: Understand the parameter count.
Original W had 2 * 3 = 6 numbers.
Our trainable parameters were B (2 numbers) and A (3 numbers). Total = 5 numbers.
In this small example, we saved only 1 number. But notice that Delta_W is a 'low-rank' matrix. Its rows are proportional to each other. The second row is exactly twice the first row. This structure is what allows us to use fewer numbers to describe the change.

If we had updated all 6 numbers in W directly, we would have needed 6 trainable parameters. By using LoRA with R=1, we used only 5. As the matrices get larger (e.g., 4096x4096), the savings become massive because R stays small while C and D grow.


## From scratch, in Python

```python
"""
Plain Python to illustrate LoRA's core computation.
No frameworks, no numpy - just lists and loops.
"""

# --- Python building blocks used ---
# A list holds things in order and can be changed: [1, 2, 3]
# A function is a reusable block of code that takes inputs, does something, and returns a result.
# A loop repeats a block of code for each item in a list.

# Step 1: Define a small frozen weight matrix W (2 rows x 3 columns)
W = [
    [1, 2, 3],
    [4, 5, 6]
]

# Step 2: Define the low-rank adapter matrices A and B
# A is 2 rows x 1 column, B is 1 row x 3 columns, rank r = 1
def make_matrix(rows, cols, value):
    """Create a matrix (list of lists) filled with a given value."""
    return [[value for _ in range(cols)] for _ in range(rows)]

A = make_matrix(rows=2, cols=1, value=0.5)  # 2x1
B = make_matrix(rows=1, cols=3, value=0.2)  # 1x3
# Note: In real training, A and B are initialised randomly and then learned.

# Step 3: Compute the outer product of A and B to get a 2x3 matrix
def outer_product(A, B):
    """Multiply A (m x 1) by B (1 x n) to get an m x n matrix."""
    result = []
    for i in range(len(A)):        # for each row in A
        row = []
        for j in range(len(B[0])): # for each column in B
            # Each cell is the product of A[i][0] and B[0][j]
            value = A[i][0] * B[0][j]
            row.append(value)
        result.append(row)
    return result

# Step 4: Scale the outer product by alpha / r
alpha = 1.0
r = 1
scale = alpha / r

def scale_matrix(M, factor):
    """Multiply every element of a matrix by a constant."""
    return [[element * factor for element in row] for row in M]

delta_W = scale_matrix(outer_product(A, B), scale)

# Step 5: Add the adapter to the frozen weights (element-wise)
def add_matrices(X, Y):
    """Add two matrices of the same shape."""
    result = []
    for i in range(len(X)):
        row = []
        for j in range(len(X[0])):
            row.append(X[i][j] + Y[i][j])
        result.append(row)
    return result

W_new = add_matrices(W, delta_W)

print("W (frozen):")
for row in W:
    print(row)
print("\nDelta (adapter):")
for row in delta_W:
    print(row)
print("\nW_new = W + delta:")
for row in W_new:
    print(row)
```


## The same thing, with a framework

```python
# Using PyTorch (typical framework for LLM work)
import torch

# Define dimensions
in_features = 3
out_features = 2
r = 1

# Frozen base weights - these never get gradients
# (torch.Tensor is like a list of numbers, but knows how to do maths on GPUs)
W = torch.tensor([[1., 2., 3.], [4., 5., 6.]], requires_grad=False)

# LoRA adapter: two small matrices with requires_grad=True so they are learned
A = torch.nn.Parameter(torch.randn(out_features, r) * 0.01)  # shape (2, 1)
B = torch.nn.Parameter(torch.zeros(r, in_features))          # shape (1, 3)

# Scale factor (alpha / r)
alpha = 1.0
scale = alpha / r

# Forward pass: y = W @ x + (alpha/r) * (A @ B) @ x
# '@' is matrix multiplication; it does exactly the loop we wrote by hand
def forward(x):
    base = x @ W.t()          # multiply input by frozen base
    adapter = x @ B.t() @ A.t()  # the low-rank correction
    return base + scale * adapter

# In a real PyTorch training loop, only A and B are optimised;
# W has requires_grad=False so the optimiser never touches it.
```


## What the code does

The plain Python builds the concept up from nothing. First I create lists-of-lists for the frozen W and the two tiny adapter matrices A and B. There are three helper functions: `make_matrix` creates a matrix filled with a value; `outer_product` loops over each row of A and each column of B, multiplying single numbers together to build the 2×3 result; `scale_matrix` multiplies every element in a matrix by a constant. Then `add_matrices` does element-wise addition, looping over positions and adding the two source values to produce the new matrix. Finally I print W, delta_W, and W_new, showing how the adapter shifts the frozen weights without ever modifying W itself.

The framework version uses PyTorch tensors (which are like lists of numbers that know how to do fast arithmetic). I set `requires_grad=False` on W so the optimiser won't update it — that's the 'frozen' part. A and B are created with `requires_grad=True` (default for `torch.nn.Parameter`) so they are the only trainable values. The forward pass does the same multiplication and addition, but using PyTorch's `@` operator for matrix multiplication, which runs the same loop we had but in highly optimised C++ on a GPU. The walkthrough emphasises that the library call is a compressed version of the hand-written loops.


## Common misconceptions

**LoRA is a small separate model** — LoRA is a little tiny neural network that I train separately, and then I plug it in like a USB stick.

LoRA is not a separate model; it modifies the effective weights of the same model by adding a low-rank correction. There is no extra network architecture; it is just a mathematical trick to approximate a large weight change with two smaller matrices. In fact: LoRA is a way to update the original model's weights. You train two small matrices and add their product to the frozen weights (or add the output to the base output). The model runs exactly the same computations, just with slightly different weights.

**LoRA reduces model quality significantly** — Because I only train a few million parameters, the model must be much worse than fully fine-tuning billions.

Research and practice show that LoRA often matches full fine-tuning quality for many tasks, especially with moderate rank (e.g., 8-16) and when the base model is already good. The low-rank assumption holds for many real-world adaptations — the change in weights needed to specialise often lives in a small subspace. In fact: LoRA can achieve near-identical performance to full fine-tuning on many tasks, and because it regularises (fewer parameters), it can even generalise better.

**QLoRA is lossier than LoRA and therefore worse** — Quantisation destroys information, so QLoRA's model must be worse than LoRA's.

QLoRA quantises the frozen base, but the adapter matrices are trained in full precision, and the quantisation error can be compensated during training. In practice, QLoRA results are very close to LoRA, often within 1-2% accuracy, while using far less memory. In fact: QLoRA is a memory-saving technique; the quantisation is only applied to the frozen weights, and the adapters learn to compensate, so the final model's performance is nearly identical to standard LoRA.

**Rank is the number of layers to adapt** — I need to set rank to the number of layers in the model, like 12 for a 12-layer model.

Rank is the size of the inner dimension of the low-rank matrices, not the number of layers. A rank of 8 or 16 is common, regardless of how many layers the model has. In fact: Rank r controls how many trainable parameters each layer has. Higher rank = more expressive, but also more memory and more risk of overfitting. It is a hyperparameter, not a count of layers.

**The bias term in LoRA is the same as model bias** — LoRA has a bias term that I need to set to avoid model bias (unfairness).

In LoRA, when we say 'bias', we mean the trainable bias parameters in the linear layers (like the bias in a neural network neuron). This has nothing to do with societal or algorithmic bias. In fact: LoRA typically freezes bias too, or trains it in some variants. The term 'bias' in neural networks is a constant added to the weighted sum, not a fairness concept.


## In short

LoRA lets you fine-tune a huge model by training only two tiny matrices per layer, which are added to the frozen base weights. QLoRA extends this by compressing the base weights into low-precision numbers, making even a 70B model trainable on a single GPU. The main levers are rank (expressiveness) and which layers you adapt. One frozen base can serve many adapters, which is the economics behind multi-tenant serving and on-premises fintech tuning.


## Where this is used

In modern AI, LoRA and QLoRA are everywhere — they are the default way to fine-tune large language models (LLMs) without a multi-GPU cluster.

**Large language models (LLMs):** When you fine-tune an LLM like Llama or Mistral for a specific task (e.g., making it follow your instructions or answer in a particular style), LoRA is used to train only a few million parameters instead of billions. A developer typically loads the base model, freezes it, and adds LoRA matrices to the attention and feed-forward layers. The result: they can fine-tune on a single GPU in hours, not weeks. The mechanism: during training, the forward pass computes both the frozen base output and the adapter output, adds them, and only the adapter's gradients are used to update A and B. At inference, the adapter weights are often merged into the base weights (W_new = W + ΔW) so there is no extra speed cost.

**Small models on devices (TinyML, phones):** When the target is a small model running on a phone, QLoRA is used during the design phase to compress a large model to fit into the device's memory. For example, a 7-billion-parameter model quantised to 4-bit (QLoRA) takes about 4 GB of memory, which can run on a modern smartphone. The developer then fine-tunes with QLoRA on a server or even on-device to adapt the model to the user's personal context, and the resulting small adapter is pushed over the air. This makes on-device personalisation practical.

**Retrieval-Augmented Generation (RAG):** LoRA is used to adapt the embedding model or the language model to your specific document corpus. For instance, you might fine-tune a lightweight embedding model with LoRA so that your internal documents get better similarity scores. The mechanism: the embedding model's weights are frozen, and a LoRA adapter is trained on pairs of similar/dissimilar documents to adjust the vector space. This improves retrieval quality without retraining the full model.

**Agentic systems that plan and call tools:** Agent systems often use a small, fast model fine-tuned with LoRA to power the 'decider' that chooses which tool to call next. The base model (e.g., a 7B model) is frozen, and a LoRA adapter is trained on trajectories of tool calls, learning when to call a calculator, when to search the web, and when to answer directly. The developer loads one base model and swaps adapters for different domains (finance, medicine, coding) — this is the multi-tenant serving pattern from your notes: one base in memory, small adapters per tenant.

**Everyday applied ML (recommendation, vision, forecasting, fraud, speech):** LoRA applies to any neural network with weight matrices, not just language models. In recommendation systems, LoRA adapts the user and item embeddings to a new domain (e.g., a new language or a new product category). In computer vision, it fine-tunes the last layers of a vision transformer on a small set of new classes without forgetting old ones. In forecasting, it adapts a pretrained time-series model to a new client's data. In fraud detection, QLoRA lets you fine-tune a large pretrained transformer on tabular data inside a bank's private cloud, satisfying data residency rules — the exact scenario in your 'QLoRA for Finance' notes.

**What a developer writes:** A developer usually reaches for the Hugging Face `PEFT` library. They load the base model and wrap it with `get_peft_model`, which automatically replaces selected layers with LoRA versions. The script is barely different from standard fine-tuning: define a training loop, pass your chat-formatted data, and the optimiser only updates the adapter. What breaks when it is done badly: choosing too high a rank wastes memory and can overfit; choosing too low can underfit. Setting the target modules to the wrong layers (e.g., all of them) slows training without benefit. Forgetting to apply the adapter during inference (not merging or not loading it) returns the base model's behaviour, which silently degrades quality — this is a common production failure.


---

Read this on the web: https://basavapurushottam.com/learn-ai/lora-and-qlora
