# Local Model Deployment

Running models on your own hardware rather than calling an API, for privacy, cost predictability or offline operation.

**Rests on:** Pre-Trained Model Adaptation

**Leads to:** Edge and Air-Gapped Inference

*LLMs and Generative AI · Inference Optimization*


## Why learn this

You want to run a large language model like the ones behind ChatGPT on your own laptop or desktop, without sending your prompts to a company's servers. This concept gives you the practical skills to do exactly that: you will learn what a model actually is as a file on disk, why it might not fit in your computer's memory, and how to use a tool called Ollama to download, run, and query a model locally. Once you can do this, you unlock the path to deeper topics in your library: understanding **quantisation** (a process of shrinking a model’s precision so it uses less memory), **inference optimization** (methods to make the model respond faster), and eventually **edge and air-gapped inference** for devices where the internet never reaches.


## The intuition

Imagine you've written a long story and want to keep it in your pocket. You could carry the full manuscript, but it's heavy. Or you could condense it to a few pages of summary, losing a little detail, but now it fits easily. A large language model is like that manuscript—a huge file on a server somewhere, containing learned patterns from terabytes of text. When you use ChatGPT, you're not carrying the file; you're sending a request to a computer that has it, and it sends back an answer. Local deployment means copying that file onto your own machine and running the model there.

Models are enormous—a 7-billion-parameter model (a model where 'parameter' refers to the adjustable settings learned during training) takes up several gigabytes of memory. Your computer might have 8, 16, or 32 GB of RAM. If the model doesn't fit, you have two choices: find a smaller model (like a pocket-sized summary) or compress the one you have. Quantisation is that compression; it reduces the number of bits used to represent each part of the model, making it smaller but potentially slightly less precise.

Once it fits, you can run it as a local service. This is a program on your machine that listens for prompts and returns answers, just like a tiny version of ChatGPT that lives inside your computer. To understand why this matters, consider 'batching' (grouping multiple requests together to process them simultaneously) and 'caching' (saving previous results so you don't have to recalculate them). These techniques help the local program use resources efficiently. If you need privacy or work without internet, 'edge inference' refers to running the model on a local device rather than a central server, while 'air-gapped inference' means doing so on a machine that has never been connected to the internet.


## An analogy

Think of ordering food from a restaurant versus cooking at home. Ordering from a restaurant (calling an API, which stands for Application Programming Interface—a set of rules allowing different software programs to talk to each other) is convenient: you pick up the phone, tell them your dish, they cook it in their kitchen with their equipment, and send it to you. You don't worry about ingredients, stove, or chef skill—but you pay per meal, you can't change the recipe, and if the restaurant goes offline, you dine in the dark.

Cooking at home (local deployment) means buying the recipe book (the model weights, which are the actual stored values the model uses to make predictions), having the right pots and pans (enough RAM and a GPU, or Graphics Processing Unit, a type of processor good for parallel calculations), and doing the cooking yourself. The first time is fiddly, but after you've done it, you can cook any time, you customise the recipe, you eat offline, and the marginal cost of each meal is just electricity. The analogy breaks down here: a restaurant can cook dishes you've never seen (the API model is often smarter), and a home kitchen is limited by its utensils (your hardware's memory and speed). So local deployment is a tradeoff: you gain privacy (your data never leaves your home), cost predictability (no per-token billing), and offline use, but you lose the capability ceiling and you take on the responsibility of maintaining your own 'kitchen'—installing software, ensuring enough memory, and troubleshooting when something breaks.


## Definition

Local model deployment is the act of installing a trained machine learning model onto your own hardware (a laptop, desktop, or single-board computer) and running it there, so that you can send prompts and receive outputs entirely without internet access or a third-party API.


## How it connects

This concept sits at the point where machine learning meets practical, everyday software. In your library's structure, it belongs to **Inference Optimization**—the parent topic that covers quantisation, distillation, batching, and caching. You have notes on **Quantization**, which is the most immediate tool for making a model fit locally; **Edge and Air-Gapped Inference**, which extends local deployment to devices that must never touch the internet; and **Latency and Speedups**, which becomes relevant when you want your local model to feel fast.


## The mathematics

The mathematics you need here is not calculus or algebra—it is **arithmetic**, the branch dealing with adding, subtracting, multiplying, and comparing quantities. The core operation is **converting between units of memory**: a byte is the basic unit of digital storage, roughly enough to hold one letter. A kilobyte (KB) is 1,000 bytes, a megabyte (MB) is 1,000 KB, and a gigabyte (GB) is 1,000 MB. In reality, computers use powers of two, but for this purpose, treat 1 GB as 1,000 MB. You also need to multiply: a model's size in bytes is (roughly) the number of parameters (weights) multiplied by the number of bytes per parameter. For example, a 1-billion-parameter model stored with 4 bytes per parameter takes about 4 billion bytes, which is 4 GB. That's a big number, but it's just multiplication and conversion. When a model doesn't fit in your computer's RAM, you can reduce the bytes per parameter by using quantisation—storing each weight with fewer digits, like rounding 3.14159 to 3.14. That's subtraction of information, but the arithmetic is still simple: fewer bytes per parameter times the same number of parameters gives a smaller total.

The memory requirement for a model is:

Model size (bytes) = (number of parameters) × (bytes per parameter)

Where:
- **number of parameters** is the count of all weights and biases in the model, often given in billions (e.g., 7B).
- **bytes per parameter** depends on the 'precision' you choose: `float32` uses 4 bytes, `float16` uses 2, `int8` uses 1, and `int4` uses 0.5 (packed).

For this concept, you can use this formula to estimate whether a model fits in your available RAM: if the product is less than or equal to your RAM, it fits; otherwise, you need to quantise or choose a smaller model.


## A worked example

```
Your laptop has 8 GB of RAM (random access memory). The model you want to run is called 'Mistral-7B', which has 7 billion parameters. The original model uses 4 bytes per parameter. You want to know if it fits.

Step 1: 7 billion parameters × 4 bytes per parameter = 28 billion bytes   # This is the total memory needed if every weight is stored as a 4-byte number.
Step 2: 28 billion bytes ÷ 1,000,000,000 bytes per GB = 28 GB   # Convert to gigabytes so you can compare with your 8 GB of RAM.
Step 3: 28 GB > 8 GB, so the original model does NOT fit   # You cannot load the whole model into memory because it's too big.
Step 4: 28 billion bytes ÷ 2 bytes per parameter = 14 GB (still too big)   # If you quantise to 2 bytes per parameter, it halves, but still not enough.
Step 5: 28 billion bytes ÷ 1 byte per parameter = 7 billion bytes = 7 GB   # With 1 byte per parameter (8-bit quantisation), it fits within 8 GB with some room for traffic.

Result: You can run Mistral-7B locally only if you quantise it to 8-bit precision, which uses 1 byte per parameter, bringing the size down to about 7 GB. That fits in your 8 GB RAM, leaving 1 GB for the operating system and the rest of your programs.
```


## From scratch, in Python

```python
# Local Model Deployment - From Scratch
# We simulate what a local model does, without any frameworks.
# This is NOT how you deploy a real LLM, but it shows the core idea.

# Imagine the model's weights are a list of numbers.
# In a real LLM, these would be millions of floats, but here we use a tiny example.
weights = [0.5, -0.2, 0.1]  # pretend these are the model's learned parameters

# The input prompt (simplified as numbers, but in a real model this would be token embeddings)
prompt = [1.0, 2.0, 3.0]

# Step 1: Compute the raw output (like a neuron's weighted sum)
def predict(input_values):
    total = 0
    for i in range(len(input_values)):
        total += input_values[i] * weights[i]
    return total

result = predict(prompt)
print("Raw output:", result)  # This is what the model would give as a number

# Step 2: This is just a dummy output. Real models output token probabilities.
# For deployment, you would load the weights from a file and use them to generate text.

# To make it a 'service', you could run this in a loop:
while True:
    user_input = input("Type something: ")
    # In reality, you'd convert text to numbers using a tokenizer, then run the model.
    # For now, we just pretend the input is a list of numbers.
    numbers = [1.0, 2.0, 3.0]  # placeholder
    print("Model response:", predict(numbers))
```


## The same thing, with a framework

In practice, you won't write your own LLM. You'll use Ollama, which is a tool made for exactly this. You install it, then you run a command like:

```bash
ollama run llama3.2:3b
```

This downloads the model (quantised to fit) and starts a local server. Then you can call it from Python using a library like `requests` or `ollama`.

Using Python's `ollama` library:

```python
import ollama

response = ollama.chat(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "What is the capital of India?"}]
)
print(response["message"]["content"])
```

Underneath, this sends an HTTP request to `http://localhost:11434`, which is the local server Ollama runs. That server loads the model into memory, processes your prompt, and returns the generated text. The model itself is stored as a file on your disk, typically in `~/.ollama/models`.


## What the code does

Let's walk through the from-scratch code first. The line `weights = [0.5, -0.2, 0.1]` creates a `list`—a collection of numbers in order. A list is like a row of boxes, each holding a value. We use `weights` to store a tiny model's parameters. The `prompt` list holds the input numbers. The `predict` function is where the action happens: it defines a reusable block of code that takes a list `input_values`, loops over each element using `for i in range(len(input_values))`, and for each i, multiplies the input by the corresponding weight and adds it to `total`. This is the same as the weighted sum in the mathematics section—just done one step at a time. The final `while` loop demonstrates serving: it keeps asking for input and printing a response, like a basic CLI chatbot. In the real world, you wouldn't implement the model yourself; you'd use Ollama. The framework code shows that: you call `ollama.chat` with the model name and a list of `messages`, where each message is a `dict` (a collection of key-value pairs) with `role` and `content`. That dict is a bit like a library index card: you look up by label, not by position. The response is also a dict, and you access `response["message"]["content"]` to get the text.


## Common misconceptions

**Local deployment is always slower than using an API.** — Because the API runs on powerful cloud servers, my laptop can never be as fast.

It's true that a huge data centre GPU is faster than a laptop CPU. But local deployment can be plenty fast for interactive use. Small models quantised to 4-bit can run at tens of tokens per second on a modern Mac or gaming PC, which feels instantaneous. API latency includes network round-trip time, which can add hundreds of milliseconds; locally, that's zero. Speed depends more on your hardware and model size than on location. In fact: Local deployment can be faster for small models and short requests, especially on good hardware, because you avoid network latency. It's not automatically slower.

**You need a powerful GPU to run any LLM locally.** — Only people with $2,000 graphics cards can run local models.

Modern laptops with 16 GB of RAM can run 3B parameter models smoothly, even on CPU only. Ollama's summary notes: quantisation makes models fit in a few GB, and models like Llama 3.2 3B run fine on an 8 GB laptop. Low-end hardware can still handle small models, and there are even models that run on Raspberry Pi. In fact: You need a machine with enough RAM (typically 8–16 GB) to fit the quantised model and some overhead. A GPU is helpful for large models, but not required for small ones.

**Local models are identical to cloud models just smaller.** — If I download Llama 3.1 8B, it will be exactly as intelligent as the ChatGPT I use online.

Cloud models like GPT-4 or Claude are typically much larger (hundreds of billions of parameters) and better aligned. Even if they share a name, open-source models like Llama 3 are not the same as proprietary ones. Your library's notes mention the 'capability ceiling'—local models may struggle with complex reasoning or rare languages. In fact: Local models are capable but often less intelligent than top-tier cloud models. You may need to adjust your expectations and sometimes use a hybrid approach.

**Once you quantise a model, it becomes useless.** — Quantisation throws away so much precision that the model's outputs become garbage.

Tests show that 8-bit quantisation loses almost no quality, and even 4-bit loses only a small amount for many tasks. The paper 'LLM.int8()' showed 8-bit inference without degradation. In practice, quantised models often outperform their full-precision selves on some metrics due to regularisation. The loss is measurable but usually modest. In fact: Quantisation trades a tiny bit of quality for significant memory savings. Many models run fine in 4-bit, and this is the standard practice for local deployment.


## In short

Local model deployment is about running an LLM on your own hardware instead of calling an API. You must check whether the model fits in your RAM, and if not, use quantisation to shrink it. Tools like Ollama make this straightforward: you download a model, run it as a local server, and call it from your code. The tradeoff is a lower capability ceiling in exchange for privacy, offline use, and predictable costs. Always start with a small quantised model, test it for your use case, and move up only if the quality is insufficient.


## Where this is used

Large language models (LLMs) are the most common target for local deployment. You might run a model like Llama 3 or Mistral on your laptop for private tasks—writing emails, summarising documents, or coding assistance—where sending the contents to a cloud server is undesirable. Developers building tools that work offline, such as a note-taking app that uses an LLM to generate tags, integrate Ollama or llama.cpp directly so the app ships with a bundled local model. Small language models (SLMs), like Phi-3.5-mini or TinyLlama, are designed specifically for this: they run on a phone or a Raspberry Pi, so you can have a voice assistant that never phones home. Retrieval-augmented generation (RAG) often runs locally too: you index your own documents (like a personal knowledge base) into a vector database, and then query it with a local LLM. The retrieval step uses embeddings and similarity search, which is lightweight, and the generation step uses the local model, keeping your data entirely on your device. Agentic systems—programs that plan, call tools (like accessing a calendar or sending an email), and act in a loop—can run locally for privacy, especially in a corporate setting where you want the agent to access internal tools without leaking secrets. Everyday applied machine learning, such as a chatbot for your local library that answers queries about opening hours, or a vision model that counts objects on a production line, benefits from local deployment for cost control and low latency on-site. What does building software with this look like in practice? A developer writes a small web service in Python that loads the model via Ollama's API or directly with PyTorch, then serve it with Flask or FastAPI. They would write something like:

```python
from flask import Flask, request, jsonify
import ollama

app = Flask(__name__)

@app.route('/chat', methods=['POST'])
def chat():
    prompt = request.json['prompt']
    response = ollama.chat(model='llama3.2:3b', messages=[{'role':'user', 'content': prompt}])
    return jsonify({'reply': response['message']['content']})
```

What breaks when it's done badly? Common failure points include choosing a model too large for the hardware, leading to out-of-memory crashes or severe swapping; neglecting quantisation, so the model runs at a snail's pace; and failing to handle concurrency—Ollama queues requests by default, so one user's prompt can block another's for minutes. Also, local models may be less capable than the cloud model you're used to, so you must set user expectations and perhaps implement a fallback to an API for difficult prompts. In summary, local deployment shines in privacy-sensitive, offline, or high-volume scenarios, but it demands that you understand the memory math and the operational responsibility that comes with running your own inference server.


---

Read this on the web: https://basavapurushottam.com/learn-ai/local-model-deployment
