# PEFT

Parameter-efficient fine-tuning: the family of methods that adapt a model by training a small number of added or selected parameters.

**Rests on:** Pre-Trained Model Adaptation

*LLMs and Generative AI · Fine-Tuning*


## Why learn this

You are about to learn one of the most practical ideas in modern AI: how to take a huge model that someone else spent months and a fortune training, and adapt it to your own task — without retraining it from scratch. To understand this, you first need to know what a 'model' is. Think of a model as a massive, pre-written book of knowledge that an AI has read. The 'weights' are the specific settings inside that book that determine how the AI processes information. 'Fine-tuning' means teaching that existing book new tricks, rather than writing a whole new book.

Understanding PEFT (Parameter-Efficient Fine-Tuning) will let you fine-tune a model on your own laptop or in your own company, even when the model has billions of parameters. A 'parameter' is a single number inside the model that controls its behavior. Changing all them takes enormous computing power. PEFT changes only a few, which is cheap and fast. This unlocks the **Fine-Tuning** parent concept and all its children: **LoRA and QLoRA** (the most popular PEFT methods), **Instruction Tuning** (making a model follow commands), **Catastrophic Forgetting** (the risk you are avoiding by using PEFT), and **Multi-Tenant Adapter Serving** (using one base model for many customers). With PEFT, you will be able to build systems that are cheap, fast, and private — and you will understand what every AI engineer means when they talk about 'adapters' and 'frozen weights'.


## The intuition

Imagine you are a manager in a large company. The company has a huge, well-trained workforce — thousands of people who can do many tasks. This workforce is like a massive model that has already learned general skills: how to write emails, how to analyze data, how to speak politely.

Now the company gets a new project: handling customer complaints in a new, very technical domain (say, legal contracts). Training the entire workforce from scratch would take years and cost a fortune. Instead, you try something smarter: you keep the existing workforce, but you give each team a small 'adjustment manual' — a few pages of instructions that tell them how to apply their general skills to this specific domain. The manual is tiny compared to the full training, but it changes the output just enough.

In AI terms, the 'existing workforce' represents the model's original knowledge. You do not change the workers themselves; you let them keep all their original skills. This is what 'frozen weights' means: the core settings of the model are locked in place so they don't forget what they already know. The 'adjustment manual' is a small set of new parameters (numbers) that you train specifically for legal contracts. You train only these new parameters, not the whole model. The model keeps all its original knowledge — it still knows English, grammar, and general facts — but it also learns your specific domain. And because the manual is small, you can swap it out, share it, or store many different manuals for different tasks. That is PEFT in a nutshell.


## An analogy

Think of a chef who has trained at a famous culinary school. The chef knows a thousand recipes — French, Italian, Indian, etc. This chef’s deep knowledge of cooking techniques (chopping, sautéing, balancing flavors) is like the 'weights' of a pre-trained model: vast and foundational.

Now a new restaurant opens: it specialises in Gujarati thali. Retraining the chef from scratch would mean sending them back to school for years. Instead, the restaurant gives the chef a small recipe card — just a few key adjustments: use this particular spice blend, this style of dal, this proportion of vegetables. The chef still knows how to chop, sauté, and plate — all the basics — but the recipe card directs their skills to the new cuisine.

In technical terms, the 'recipe card' is an 'adapter.' It is a tiny set of new parameters added to the system. You train only this card, not the chef’s entire brain. If the restaurant later wants to serve Punjabi food, they just give the chef a different recipe card. The chef's core knowledge stays the same; only the small set of instructions changes.

This is exactly how PEFT works: you take a pretrained model (the chef), freeze its weights (the chef's core knowledge), and add a tiny set of new parameters (the recipe card) that you train on your specific task. The new parameters are so small that they can be versioned, stored, and swapped like configuration files.

And here is where the analogy stops: a human chef can only hold one recipe card in mind at a time, but a model with PEFT can have many adapters loaded dynamically — thousands of different 'recipe cards' can be attached to the same base model, and you can switch between them in milliseconds.


## Definition

Parameter-efficient fine-tuning (PEFT) is the family of methods that adapt a pretrained model to a new task by training a small number of added or selected parameters, while freezing the original (base) weights to preserve the model's general capabilities.


## How it connects

You are at the start of the **Fine-Tuning** branch. Fine-tuning is the general practice of taking a pretrained model and adjusting it to a new task. PEFT is the most efficient way to do that. Your library notes mention that **LoRA and QLoRA** are the most popular PEFT methods, which use tiny mathematical tweaks to achieve this efficiency. **Multi-Tenant Adapter Serving** is what makes PEFT economically powerful: one frozen base model can serve many customers, each with their own small adapter. PEFT also directly addresses the **Catastrophic Forgetting** risk — because you are not changing the base weights, you do not lose the original abilities. Later concepts like **Instruction Tuning** and **Classification Fine-Tuning** can be done with PEFT methods.


## The mathematics

PEFT (Parameter-Efficient Fine-Tuning) relies on a simple idea from **linear algebra** — the branch of mathematics that deals with tables of numbers, called matrices.

A deep learning model is essentially a stack of these tables. Each table contains *parameters* (weights and biases). When you train a model normally, you adjust every single number in every table. PEFT says: do not touch the original numbers; instead, add a few new numbers *in a clever way*. These new numbers are usually arranged as **low-rank matrices**.

To understand this, we must first define what a matrix is. A matrix is just a rectangular grid of numbers. It has rows (horizontal lines) and columns (vertical lines). We describe its size by saying 'rows by columns'. For example, a 2-by-3 matrix has 2 rows and 3 columns.

What is *rank*? The rank of a matrix tells us how much independent information it holds. A 'low-rank' matrix is one that can be built from very little information. Specifically, any low-rank matrix can be created by multiplying two smaller matrices together. This is the key trick of **LoRA** (Low-Rank Adaptation), the most common PEFT method.

Why do we do this? Imagine a large weight matrix in a model that is 1000 rows by 1000 columns. It contains 1,000,000 numbers. If we want to change this matrix slightly during fine-tuning, updating all 1,000,000 numbers requires a lot of memory and computation.

Instead, LoRA replaces the update with two tiny matrices. Let us say we choose a 'rank' of 10. We create:
1. Matrix A: size 10 by 1000 (10 rows, 1000 columns). This has 10,000 numbers.
2. Matrix B: size 1000 by 10 (1000 rows, 10 columns). This has 10,000 numbers.

When we multiply these two small matrices together (A times B), the result is a matrix of size 1000 by 1000. This resulting matrix represents the *change* we want to make to the original weights. Because we only trained the numbers in A and B, we only updated 20,000 parameters instead of 1,000,000. That is a 50-fold reduction in trainable parameters.

The mathematics you need to follow this logic is just two operations:
1. **Matrix Multiplication**: Combining two tables of numbers to produce a new table. If you multiply an (m-by-k) matrix by a (k-by-n) matrix, the result is an (m-by-n) matrix. Each number in the result is the sum of products from a row of the first matrix and a column of the second.
2. **Matrix Addition**: Adding two tables of numbers together. You simply add the number in the first row, first column of the first table to the number in the first row, first column of the second table, and so on for every position. This only works if both tables have the exact same shape.


## A worked example

Let us walk through a concrete example with small numbers.

**The Setup**
Imagine a very small layer in a neural network.
- The input is a vector (a table with 1 column) of size 3-by-1. Let us call this input **x**.
- The original weight matrix is **W**. It has 2 rows and 3 columns (size 2-by-3). This means it takes an input of 3 numbers and produces an output of 2 numbers.

Let us assign specific values to these tables:
- Input **x** = [[1], [2], [3]]
- Original Weight **W** = [[1, 0, 2], 
                        [0, 1, 1]]

Without PEFT, the output of this layer is calculated by multiplying W by x. But we are going to use LoRA to adjust W.

**The LoRA Adjustment**
We want to add a small change, which we will call **ΔW** (Delta-W), to the original weight matrix W. The new effective weight matrix will be called **W_eff**.

Formula: W_eff = W + ΔW

To keep the number of trainable parameters low, we do not create ΔW directly. Instead, we create two small matrices, **A** and **B**, and multiply them to get ΔW.

Formula: ΔW = A × B

We choose a rank **r = 1**. This means:
- Matrix **A** will have 1 row and 3 columns (size 1-by-3). Wait, let us check the dimensions carefully. To multiply A and B to get a 2-by-3 matrix (the shape of W), we need:
  - A to be size (rows of W) by r. So A is 2-by-1.
  - B to be size r by (columns of W). So B is 1-by-3.

Let us assign values to A and B:
- Matrix **A** (2-by-1) = [[2], [1]]
- Matrix **B** (1-by-3) = [[1, -1, 0]]

**Step 1: Calculate ΔW by multiplying A and B**
We multiply the 2-by-1 matrix A by the 1-by-3 matrix B. The result is a 2-by-3 matrix.

The rule for multiplication here is simple because A has only one column and B has only one row. We take each number in A and multiply it by every number in B.

- First row of ΔW comes from the first number in A (which is 2):
  - 2 × 1 = 2
  - 2 × (-1) = -2
  - 2 × 0 = 0
  So the first row is [2, -2, 0].

- Second row of ΔW comes from the second number in A (which is 1):
  - 1 × 1 = 1
  - 1 × (-1) = -1
  - 1 × 0 = 0
  So the second row is [1, -1, 0].

Thus, **ΔW** = [[2, -2, 0], 
                [1, -1, 0]]

Note that we only had to train/store 4 numbers here (the two numbers in A and the three numbers in B? No, A has 2 numbers, B has 3 numbers. Total trainable parameters = 2 + 3 = 5). In this specific rank-1 case, we used 5 parameters to define a full 2-by-3 matrix change.

**Step 2: Calculate W_eff by adding ΔW to W**
Now we add the change matrix ΔW to the original frozen weight matrix W. We add them element by element (top-left to top-left, etc.).

Original **W** = [[1, 0, 2], 
                  [0, 1, 1]]

Change **ΔW** = [[2, -2, 0], 
                 [1, -1, 0]]

Addition:
- Top-left: 1 + 2 = 3
- Top-middle: 0 + (-2) = -2
- Top-right: 2 + 0 = 2
- Bottom-left: 0 + 1 = 1
- Bottom-middle: 1 + (-1) = 0
- Bottom-right: 1 + 0 = 1

Thus, **W_eff** = [[3, -2, 2], 
                   [1, 0, 1]]

**Result**
The adapted layer now uses **W_eff** instead of **W**. During inference (when the model is running), we can simply add the matrices together once and store the result, or keep them separate. The key point is that during training, we only updated the 5 numbers in A and B. We did not touch the 6 numbers in W. In a real large model, W might have millions of numbers, while A and B might have only thousands, leading to massive savings in memory and computation.


## From scratch, in Python

```python
import random

# We'll simulate a tiny model: one linear layer with input size 3 and output size 2.
# The weight matrix W is 2 rows (outputs) and 3 columns (inputs).
W = [[1, 0, 2],
     [0, 1, 1]]

# A function that takes an input vector and a weight matrix, and returns the output.
# This is the forward pass: for each output row, multiply each input value by
# the corresponding weight and add them up.
def forward(x, weight_matrix):
    output = []
    for row in weight_matrix:
        total = 0
        for i in range(len(x)):
            total += row[i] * x[i]
        output.append(total)
    return output

# The adapter: we choose a small 'rank' r. Here r = 1.
# We create two small matrices A (2x1) and B (1x3).
# In real code we would initialise them randomly, but for clarity use simple numbers.
A = [[2], [1]]
B = [[1, -1, 0]]

# Compute the change matrix ΔW = A * B (matrix multiplication).
# Since A is 2x1 and B is 1x3, the result is 2x3.
def matrix_multiply(X, Y):
    # X: m x n, Y: n x p, result: m x p
    m = len(X)
    n = len(X[0])
    p = len(Y[0])
    result = [[0] * p for _ in range(m)]
    for i in range(m):
        for j in range(p):
            for k in range(n):
                result[i][j] += X[i][k] * Y[k][j]
    return result

delta_W = matrix_multiply(A, B)  # [[2, -2, 0], [1, -1, 0]]

# Add delta_W to W to get the effective weight matrix.
def add_matrices(X, Y):
    # Both are m x n
    result = []
    for i in range(len(X)):
        row = []
        for j in range(len(X[0])):
            row.append(X[i][j] + Y[i][j])
        result.append(row)
    return result

W_eff = add_matrices(W, delta_W)

# Test with an input.
x = [1, 2, 3]
print('Output using W:', forward(x, W))
print('Output using W_eff:', forward(x, W_eff))

# In a real training loop, we would update only A and B (the adapter) to minimise
# a loss, while keeping W frozen.
```


## The same thing, with a framework

In PyTorch, you would use the `peft` library. The code looks like this:

```python
from peft import LoraConfig, get_peft_model, TaskType
import torch

# Assume 'model' is your pretrained model (e.g., a transformer).
# Define the LoRA configuration: we will adapt the query and value projections.
config = LoraConfig(
    task_type=TaskType.SEQ_2_SEQ_LM,
    r=8,              # rank
    lora_alpha=32,
    target_modules=["q", "v"],
    lora_dropout=0.1,
)

# Wrap the model with the adapter.
peft_model = get_peft_model(model, config)

# Now train peft_model on your dataset. Only the adapter parameters will be updated.
# The base model's weights stay frozen. This is what the 'from_scratch' code did manually.
```


## What the code does

The from-scratch code shows the essential idea of LoRA. First, it defines a tiny model with a weight matrix W. Then it creates two small matrices A and B (the adapter). The `matrix_multiply` function computes their product, giving a change matrix ΔW. The `add_matrices` function adds this change to W to get the effective weights. The `forward` function computes the output using either the original or the adapted weights. In real training, you would update A and B (the adapter) using gradient descent, while keeping W frozen, so the model only learns a small change. The framework code shows how the `peft` library wraps a large model and does this automatically, exactly replicating the manual process. The `r` parameter is the rank; `lora_alpha` is a scaling factor; `target_modules` says which layers to adapt.


## Common misconceptions

**PEFT means you do not train anything** — I thought PEFT just means you don't train at all — you just pick a good model and add a prompt.

PEFT still involves training — you train the added adapter parameters. You are just not training the base weights. In fact: PEFT is a kind of fine-tuning where you train only a small subset of (usually newly added) parameters, while freezing the original model weights.

**Adapter means a separate model** — I thought an adapter is a whole second model that we combine with the base model at inference.

Adapters are not separate models; they are small sets of parameters that plug into the existing layers. There is no second big network. In fact: An adapter is a small trainable module inserted into a frozen model. It modifies the forward pass only in small ways, and at runtime it is just a few extra matrix multiplications inside the existing network.

**PEFT is always worse than full fine-tuning** — I thought that because PEFT trains fewer parameters, it must always produce a worse result than full fine-tuning.

In practice, PEFT often performs just as well or even better because it preserves the base model's general knowledge and avoids catastrophic forgetting. The low-rank constraint can act as a regulariser. In fact: PEFT can match full fine-tuning on many tasks, and is often preferred for its efficiency and safety.

**You need to train the whole adapter every time** — I thought every time you want a new task, you need to retrain the adapter from scratch on the entire model.

You train the adapter on your task data, but you can start from a fresh random initialisation or from another adapter. The base model is frozen, so you only update the adapter parameters. In fact: For each new task, you train a new adapter, but it is a separate small training run, not retraining the base model.


## In short

PEFT (Parameter-Efficient Fine-Tuning) is the art of adapting a pretrained model by training a small number of added parameters, while freezing the base weights. The base model's original abilities are preserved, preventing catastrophic forgetting. Adapters are small — measured in megabytes — and can be versioned and swapped like configuration. LoRA and QLoRA are the most popular methods: they train two small matrices that approximate a change to the weight matrix. This makes fine-tuning cheap and fast, enables multi-tenant serving (one base model, many adapters), and underpins much of modern applied AI, from LLMs on the cloud to tiny on-device models.


## Where this is used

PEFT is everywhere in modern AI. Here is how it is used, with concrete mechanisms.

**Large Language Models (LLMs):** The biggest use. Models like GPT-4 or Llama have billions of parameters; full fine-tuning would need thousands of GPUs. PEFT (especially LoRA) lets a single developer fine-tune on one consumer GPU. For example, to fine-tune a 7-billion-parameter model on your own customer-support dataset, you freeze all original weights and insert LoRA adapters into the attention layers. Training updates only the adapter matrices (a few million parameters) while the base stays fixed. This is described in your library note from *Fine-Tuning with Python*: 'PEFT reframes fine-tuning as adapter training: the base model stays frozen, so a single deployment can serve many tenants' adapters.'

**Small Models on Devices:** On a phone or a smart speaker, memory is tiny. If you fine-tune a model to recognise your voice, you cannot store a full copy of the adapted model — that would be hundreds of megabytes. Instead, you store only the adapter (a few megabytes) and load it into the frozen base on the phone. This is **QLoRA** applied: you also quantise (shrink) the base model to 4-bit precision, so it fits on a phone, and the adapter is even smaller. The developer writes code similar to the `from_scratch` example but uses `peft` library to load a 4-bit base and attach the adapter. The effect: a tiny, efficient model that still adapts to you.

**Retrieval-Augmented Generation (RAG):** RAG combines a retrieval system with a generator. PEFT is used to adapt the retriever or the generator to your domain. For instance, you can fine-tune the embedding model (which converts text to vectors) with PEFT so that documents from your legal corpus are better matched. The mechanism: the embedding model has a frozen base, and you add a LoRA adapter to it that is trained to better rank your domain documents. The adapter is small, so you can have different adapters for different domains, and you can load the right one on demand.

**Agentic Systems:** Agents that plan and call tools are LLMs wrapped in a loop. You want the agent to use your company's specific tools. Full fine-tuning would be huge. Instead, you train a LoRA adapter on examples of tool calls (e.g., 'call the inventory API with these arguments'). The mechanism: the adapter changes the model's output distribution slightly, making it more likely to generate the correct API call. The base model remains general, so it still knows how to chat and reason, but the adapter steers it toward your tools. This is lightweight and cheap, so you can version adapters per client.

**Everyday Applied Work:** Recommendation systems, fraud detection, forecasts — any machine learning task that uses a neural network can benefit. Instead of retraining a large model for each new customer, you train a small adapter on that customer's data. For example, in finance, **QLoRA for Finance** (as in your library) is used to adapt a language model to financial terminology, but the data never leaves the company's controlled environment because the adapter is trained on-premises. The mechanism: the adapter learns domain-specific numbers and vocabulary, while the base model remains frozen, so sensitive data stays in-house.

**What it looks like in practice:** A developer reaches for the `peft` library in Python (with PyTorch), or the `transformers` library from Hugging Face. They write: `model = AutoModelForCausalLM.from_pretrained('base_model', load_in_4bit=True)`, then `model = get_peft_model(model, LoraConfig(r=8, ...))`. What breaks when done badly: if the rank is too small, the adapter cannot capture the domain pattern and performance is poor; if you train the adapter on too little data, it overfits and forgets the task (a form of **Catastrophic Forgetting**, even with PEFT, though less severe).


---

Read this on the web: https://basavapurushottam.com/learn-ai/peft
