# QLoRA for Finance

Applying quantised low-rank adaptation to financial models, where domain vocabulary matters and data cannot leave controlled environments.

**Rests on:** Foundation Models, LoRA and QLoRA

*LLMs and Generative AI · Fine-Tuning*


## Why learn this

This skill lets you customise a large language model for specific financial tasks without needing expensive supercomputers or sending sensitive data off your own servers. After reading this, you will be able to: (1) explain clearly what QLoRA does to a colleague or client, (2) read a training script and understand exactly what each line achieves, and (3) assess whether a proposed training plan keeps your regulated data secure. This approach is the practical key to building systems like an earnings call assistant that understands your specific jargon, using methods that are far less resource-intensive than traditional training.


## The intuition

Imagine you have a massive financial regulations handbook — thousands of pages of general knowledge about markets, reporting, and compliance. Your team now needs a version that is exceptionally good at answering questions about Indian GST rules. Rewriting the entire book would take years and risks accidentally removing useful general context. Instead, you write a thin insert of just a few pages covering only the GST specifics. You slip this insert into the back of the book. The original book remains untouched; you have added only the insert. When asked a GST question, the system checks the insert first. This is how QLoRA works. The 'book' is the pre-trained model, containing billions of numbers that determine its behaviour. The 'insert' is a small set of new numbers, known as an adapter. 

QLoRA has two key parts. First, it keeps the original book completely frozen so you don't lose its general knowledge. This prevents what we call catastrophic forgetting, which is the risk that if you change too many details in a massive system designed for broad understanding, it will stop understanding everything well. By freezing the original book, you ensure your new system still knows how to talk about markets generally while learning GST specifically.

Second, QLoRA makes the book itself take up less physical space by compressing its data. It does this through a process called quantisation. Quantisation is simply the act of reducing the precision of the numbers stored in the model — for example, changing from highly detailed 32-bit numbers to simpler, compressed 4-bit or 8-bit numbers. This compression allows you to store the entire massive model on a single laptop rather than needing a room full of servers. The trick is that you never change these compressed numbers during training; you only train the small adapter insert.


## An analogy

Think of a senior financial analyst who already knows general finance well, and who you need to specialise in detecting fraud in small business loans. You cannot send her back to university to re-learn everything; that would risk 'catastrophic forgetting', where she loses her broad expertise and becomes less effective overall. Instead, you give her a small cheat sheet with specific patterns and rules for fraud detection. She studies this sheet alongside real loan examples, adjusting only her 'fraud instincts' — a tiny part of her knowledge base — while keeping her general finance skills sharp. 

This is like Low-Rank Adaptation (LoRA). LoRA works by training only a few new numbers added to the existing model, rather than changing all the original billions of numbers. Conceptually, instead of recalculating the entire complex web of connections in her brain, you give her a simplified shortcut — a 'low-rank matrix'. Think of this low-rank matrix as a small, manageable filter that sits between her general knowledge and the specific fraud signals, allowing you to tweak just a few key pathways rather than rewiring the whole mind. 

Now, imagine this analyst’s brain is so vast that processing it requires more coffee (computing power) than your office can provide. To make it feasible on your existing laptop, you also compress her memories into a lower-resolution format. She loses a tiny bit of mental sharpness, but she remains highly capable for the task at hand. This compression step is 'quantisation'. The core idea holds: change only a small, new part, keep the old part frozen, and compress the old part to fit your hardware limits.


## Definition

QLoRA (Quantised Low-Rank Adaptation) is a technique that fine-tunes a large language model by keeping its original billions of parameters completely unchanged and compressed for efficiency, while training only a small, newly added set of parameters (the adapter) to handle specific tasks.


## How it connects

This method falls under the broader category known as Parameter-Efficient Fine-Tuning (PEFT), which refers to strategies that adapt a large model by training only a small subset of new parameters while leaving the original ones frozen. QLoRA is a specific variation of LoRA, which itself trains low-rank matrices (simplified shortcuts for calculating adjustments) added to the base weights. It is distinct from classification fine-tuning, which adds a simple output layer for categorising data, and continued pre-training, which involves feeding new raw text into the model to expand its general knowledge base rather than just adapting it for specific instructions.


## The mathematics

This concept combines two ideas: how neural networks multiply numbers together, and how we can shrink those numbers to save space without losing too much accuracy.

First, let us look at the multiplication. A neural network layer takes an input vector x and multiplies it by a weight matrix W to produce an output y. In standard notation, this is written as y = Wx. Here, W is a large table of numbers (a matrix) that holds what the model has learned. The letter x represents the data coming in, and y is the result going out.

In QLoRA, we change how this multiplication works by splitting it into two parts. We keep the original weight matrix W, but we freeze it so it never changes. Instead of using W directly during training updates, we add a small correction to it. This correction is built from two smaller matrices, which we will call A and B.

Matrix A has dimensions R by D. Matrix B has dimensions C by R. The number R is called the rank, and it is much smaller than both C and D. Because R is small, the product of these two matrices, written as BA, creates a matrix that has low rank. This means the correction it provides is simple and uses very few parameters.

The new operation becomes y = W_quant x + (BA)x. Here, W_quant is the quantized version of the original weight matrix W. The term (BA)x is the adapter's contribution. We can group this as y = (W_quant + BA)x. This shows that the effective weight applied to the input is the sum of the frozen, compressed base weights and the trainable low-rank correction.

The matrices A and B are initialized in a specific way to ensure training starts correctly. Matrix A is filled with small random numbers from a Gaussian distribution. Matrix B is set entirely to zero. Because B starts at zero, the product BA is also zero at the start. This means the initial output of the model is identical to the output of the original quantized model. As training proceeds, gradient descent updates A and B, allowing the model to learn small adjustments.

The second part of QLoRA is quantization. Quantization reduces the precision of the numbers in the weight matrix W. Instead of storing each weight as a 32-bit floating-point number, we store it with fewer bits, such as 4 bits or 8 bits. This requires rounding the original weights to the nearest representable value in the lower-precision format. For example, a weight of 0.123456 might be rounded to 0.12. These quantized weights are frozen; they do not change during training. Only the adapter matrices A and B are updated with full precision.

This approach saves memory because the large matrix W_quant takes up less space, and it saves computation time because we only need to update the small matrices A and B, which have far fewer parameters than W.


## A worked example

We will walk through a tiny example with one input feature and one output neuron. This helps us see how the numbers change.

Setup:
- Original weight W: 0.123456789
- Input x: 5
- Target output y_target: 3
- Quantization scale: We round to 1 decimal place (e.g., 0.1, 0.2, etc.).
- Rank R: 1 (so A and B are just single numbers).

Step 1: Quantize the base weight.
We take the original weight W = 0.123456789 and round it to one decimal place.
Rounding 0.123456789 to one decimal place gives 0.1.
So, W_quant = 0.1.
This value is now frozen. It will not change during training.

Step 2: Initialize the adapter matrices A and B.
The rank R is 1, so A is a single number (let's call it a) and B is a single number (let's call it b).
We initialize B to zero: b = 0.
We initialize A with a small random value. Let us say the random initialization gives a = 0.5.
Note: In the standard LoRA paper, B is initialized to zero so that the initial correction is zero. Here, we have set b=0, so the initial product BA is 0 * 0.5 = 0. Wait, if b=0, then BA=0 regardless of a. Let us follow the standard initialization: A is random, B is zero.
So, a = 0.5 (random), b = 0 (fixed initialization).
The initial correction matrix BA is the product b * a = 0 * 0.5 = 0.

Step 3: Compute the initial prediction.
The effective weight is W_effective = W_quant + BA.
W_effective = 0.1 + 0 = 0.1.
The prediction y_pred is the effective weight times the input x.
y_pred = 0.1 * 5 = 0.5.

Step 4: Calculate the error.
The target output is 3.
The error is y_target - y_pred = 3 - 0.5 = 2.5.

Step 5: Update the adapter.
We do not update W_quant. It stays at 0.1.
We update A and B using gradient descent. For this example, let us assume the gradient step results in a change to b.
Suppose the update changes b from 0 to 1.0. (In reality, this depends on the learning rate and the gradient of the loss function).
Now, b = 1.0.
a remains 0.5 (for simplicity, let us assume it also updates or stays fixed; let us say it stays 0.5 for this step).
The new correction BA is b * a = 1.0 * 0.5 = 0.5.

Step 6: Compute the new effective weight.
W_effective_new = W_quant + BA_new.
W_effective_new = 0.1 + 0.5 = 0.6.

Step 7: Compute the new prediction.
y_pred_new = W_effective_new * x.
y_pred_new = 0.6 * 5 = 3.0.

The new prediction is 3.0, which matches the target exactly. The base weight W_quant remained frozen at 0.1 (quantized), while the adapter matrices A and B adjusted to provide a correction of 0.5. This demonstrates how QLoRA learns by modifying only the small adapter components.


## From scratch, in Python

```python
import math

# ---- Python building blocks ----
# A list is an ordered collection that can be changed: e.g. my_list = [1, 2, 3]
# A function is a reusable block of code: def name(parameters): ...

# We will simulate a tiny linear layer with a frozen, quantised weight matrix.
# In real QLoRA, the weight matrix is huge, but here we use 2x2 for clarity.

# 1. Define the base weight matrix (pretrained, will be frozen and quantised)
base_weights = [
    [0.123456789, 0.987654321],
    [0.555555555, 0.111111111]
]

# 2. Quantise: round each value to one decimal place (simplified)
def quantise(matrix):
    quantised = []
    for row in matrix:
        new_row = []
        for value in row:
            new_row.append(round(value, 1))
        quantised.append(new_row)
    return quantised

quantised_weights = quantise(base_weights)
print("Quantised weights:", quantised_weights)

# 3. Define the input vector (a single data point, e.g., features of a transaction)
input_vector = [2.0, 1.0]

# 4. Define a function to compute the output of a linear layer (matrix multiplication)
def linear_layer(weights, input_vec):
    output = []
    for i in range(len(weights)):  # loop over rows
        total = 0.0
        for j in range(len(input_vec)):  # loop over columns
            total += weights[i][j] * input_vec[j]
        output.append(total)
    return output

# This is exactly what a matrix-vector product is: for each row, sum the products.
print("Output with quantised weights:", linear_layer(quantised_weights, input_vector))

# 5. Now define the adapter as two small matrices A and B (rank 1, meaning bottleneck size 1)
# A is shape (1, 2) - maps 2 inputs to 1 bottleneck
A = [[0.5, -0.2]]  # one row, two columns
# B is shape (2, 1) - maps 1 bottleneck back to 2 outputs
B = [[0.3], [0.8]]  # two rows, one column

# 6. Compute the adapter update for the input: first project down (A * input), then project up (B * that)
def mat_vec_mul(matrix, vec):
    result = []
    for row in matrix:
        total = 0.0
        for j in range(len(vec)):
            total += row[j] * vec[j]
        result.append(total)
    return result

def mat_mul(A, B):
    # A is (rows_A, cols_A), B is (cols_A, cols_B)
    rows_A = len(A)
    cols_A = len(A[0])
    cols_B = len(B[0])
    # B must have rows_A rows? Actually B here is (2,1), we need B to have shape (rows_A = 2, cols_B = 1) but we treat it as matrix multiplication
    # For simplicity, we'll manually do it for our specific shapes.
    # We want BA to be (2,2) = B * A
    BA = []
    for i in range(len(B)):
        row = []
        for j in range(len(A[0])):
            sum_ = 0.0
            for k in range(len(A)):
                sum_ += B[i][k] * A[k][j]
            row.append(sum_)
        BA.append(row)
    return BA

# Compute BA (the low-rank update matrix)
BA = mat_mul(B, A)
print("BA (low-rank update):", BA)

# 7. Add BA to the quantised weights to get the effective weights
def add_matrices(m1, m2):
    result = []
    for i in range(len(m1)):
        row = []
        for j in range(len(m1[0])):
            row.append(m1[i][j] + m2[i][j])
        result.append(row)
    return result

effective_weights = add_matrices(quantised_weights, BA)
print("Effective weights:", effective_weights)

# 8. Compute output with the effective weights
print("Output with effective weights:", linear_layer(effective_weights, input_vector))

# 9. In real QLoRA training, we would compute a loss (how wrong the output is) and
#    update ONLY A and B using gradient descent, while keeping quantised_weights frozen.
#    This small example shows the mechanics: quantisation compresses, adapter adjusts.
```


## The same thing, with a framework

In practice, you would use Hugging Face's transformers library with bitsandbytes for quantisation and peft for low-rank adapters. A typical QLoRA training script would look like:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import bitsandbytes as bnb

# Load model with 4-bit quantisation
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_4bit=True,  # quantise to 4 bits
    quantization_config=bnb.BNBConfig(load_in_4bit=True, bnb_4bit_use_double_quant=True)
)

# Prepare the model for k-bit training (freezes base, prepares for adapters)
model = prepare_model_for_kbit_training(model)

# Configure LoRA - only these small matrices will be trained
lora_config = LoraConfig(
    r=8,  # rank of the adaptation matrices
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],  # which attention layers to adapt
    lora_dropout=0.1,
    bias="none"
)

# Attach the adapters to the model
model = get_peft_model(model, lora_config)

# Now you can train with your financial dataset (already in chat format)
training_args = TrainingArguments(
    output_dir="./qlora_finance",
    per_device_train_batch_size=4,
    learning_rate=2e-4,
    num_train_epochs=3,  # a few epochs on a small budget
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=your_finance_dataset,
)
trainer.train()

# To use the fine-tuned model, you load the base model again and apply the adapter
trained_adapter = model.peft_config  # this is where the adapter weights are stored
```

Underneath, this is doing exactly what the from-scratch code did: freezing the quantised base weights, creating small A and B matrices, and adding their product to the base during each forward pass. The library handles the intricate quantisation algorithm (down to 4 bits with careful rounding) and the gradient updates to A and B only.


## What the code does

The from-scratch code shows the entire pipeline in miniature. First, we have a base weight matrix (the model's knowledge). We quantise it by rounding to one decimal place — that's the compression step. Then we define a tiny adapter made of two small matrices A and B. Multiplying A and B gives a low-rank matrix BA that is added to the quantised weights. The output is computed with these effective weights. In real training, we would compute a loss (how wrong the predictions are on your financial data) and then update A and B only, using gradient descent. The quantised base weights never change. The framework code shows how you do this with the actual libraries: load a huge model quantised to 4 bits, prepare it for training, attach a LoRA configuration, and train only the adapter weights. The library handles the complex quantisation algorithm and the gradient math; the key concept is the same as in the small example.


## Common misconceptions

**QLoRA retrains the whole model** — QLoRA updates all the weights, just faster than ordinary fine-tuning.

In QLoRA, the base model's weights are frozen and also quantised; they never change during training. Only the small adapter matrices are updated. In fact: QLoRA trains only a small set of added parameters (the adapter) while the original weights stay frozen and compressed.

**Quantisation is lossless** — If I round the weights, the model still performs exactly the same as before.

Rounding numbers loses information. The model's output changes slightly, but QLoRA is designed so that the adapter can compensate for the quantisation error during fine-tuning. In fact: Quantisation reduces precision and can slightly lower base performance; the adapter is trained to restore quality on your specific task.

**Adaptation injects new knowledge** — Fine-tuning with QLoRA is how I teach the model facts it did not know, like the latest stock prices.

Fine-tuning changes behaviour, not stored facts. The model may memorise bits of training data but it has no reliable database. For facts, use retrieval. In fact: QLoRA is for shaping behaviour (e.g., how to reason about financial text, follow instructions, call tools). Facts come from retrieval or external data.

**QLoRA is only for huge models** — I have a small model on my laptop, so QLoRA is not relevant to me.

QLoRA is also useful for small models: it lets you fine-tune a 3B-parameter model on a single consumer GPU, and it can make deployment easier via further quantisation. In fact: QLoRA helps any model where memory or compute is constrained, from 1B-parameter models on phones to 70B models on servers.

**QLoRA removes the need for data preparation** — I can just feed raw financial PDFs into QLoRA and it will learn everything.

The model needs data in a specific format — usually the chat template — and the data must be high quality and representative. QLoRA cannot fix poorly prepared data. In fact: Data shaping into the model's chat format is a necessary step before any fine-tuning, including QLoRA.


## In short

QLoRA is a way to fine-tune a large language model on your own hardware, without retraining the billions of original weights. It compresses those weights to 4 or 8 bits so they fit in memory, and trains only a small adapter (two small matrices) that are added to the quantised weights during a forward pass. This makes domain adaptation feasible for finance where data cannot leave a controlled environment. Remember: the base is frozen, the adapter learns; quantisation is compression with a trade-off; and facts are for retrieval, behaviour is for tuning.


## Where this is used

In large language models: QLoRA is the standard way to fine-tune huge models like Llama-2-70B on a single GPU. Instead of loading all 70 billion weights in full precision (which would take ~280 GB of memory), you load them in 4-bit quantisation (~35 GB) and train a small adapter (maybe 0.1% of the parameters). This makes fine-tuning feasible for smaller teams and startups. In finance, a bank can take a general-purpose model like Llama-2 and adapt it to its own internal documents without sending those documents to a third-party API — the whole process runs on premises, under the bank's firewall. That is critical because of data residency rules: financial data often cannot legally leave the country or the corporate network. QLoRA makes it possible to fine-tune a model completely on your own hardware.

In small language models running on devices: Quantisation (the Q in QLoRA) is also used for inference on phones and laptops. After a model is fine-tuned with QLoRA, you can further quantise the adapter or just keep it at full precision — the adapter is tiny. This means a bank's loan officer could have a small fine-tuned model running offline on their laptop, able to answer questions about internal policy without ever sending a query over the network. The combination of QLoRA training (quantising the base) and post-training quantisation (quantising the adapter) is the path to edge deployment.

In retrieval-augmented generation (RAG): RAG systems retrieve relevant documents and feed them to the model as context. QLoRA can fine-tune the model to better use that retrieved context — e.g., to follow instructions like 'Use only the provided paragraphs to answer.' In finance, you might have a large corpus of regulatory filings. You embed the documents, retrieve the top few for a question, and the fine-tuned model is trained to synthesise answers strictly from those, avoiding hallucination. The adapter's job is to shape the model's behaviour for that style of interaction, rather than to inject facts (that's what the retrieval does).

In agentic systems that plan and call tools: An agent might need to call APIs to fetch stock prices, check account balances, or execute trades. The base model knows how to generate text, but not necessarily how to format the JSON for a tool call. QLoRA is used to fine-tune a model on thousands of examples of 'intent → well-formed tool call → parse response → next action'. In finance, this could be an earnings call assistant that first retrieves the call transcript, then calls a function to compute financial ratios, then answers in a structured summary. The adapter learns the specific syntax and reasoning pattern. It is much cheaper than training a model from scratch, and the base model's general language ability remains intact.

In everyday applied machine learning: Outside of generative models, QLoRA-style ideas appear in recommendation systems, fraud detection, and forecasting. Any deep learning model can be quantised after training, and low-rank adaptation can be used to personalise a model for a specific customer or a specific market segment. Imagine a fraud detection model that all banks share (trained centrally on non-sensitive data) — each bank can download the base model and add a small adapter trained on its own transactions to specialise it without sharing its data. The adapter learns bank-specific patterns while the base stays common. In practice, a developer using QLoRA writes code like the framework snippet above: load the base model, prepare it for k-bit training, configure the LoRA parameters, and train. What breaks when done badly is catastrophic forgetting — if you train too long or on too narrow a dataset, the model may lose general capability and become overfit to the training distribution, or misparse formats. Also, if the quantisation is too aggressive (e.g., 2-bit without careful calibration), the base model's quality degrades so much that the adapter cannot compensate. And if you ignore the chat format template, your fine-tuned model will not respond properly to system messages, appearing weak or broken.


---

Read this on the web: https://basavapurushottam.com/learn-ai/qlora-for-finance
