# Edge and Air-Gapped Inference

Serving models on devices or in isolated networks with no external connectivity, under strict compute and update constraints.

**Rests on:** Local Model Deployment

*LLMs and Generative AI · Inference Optimization*


## Why learn this

You are learning this because it unlocks the ability to deploy artificial intelligence where the internet does not reach. Concretely, after this, you will be able to understand how models run on your phone, on a factory floor computer that is physically cut off from the outside world, or in a hospital where patient data legally cannot leave the building. This leads directly to your next topics: Local Model Deployment (running models on your own hardware) and Quantization (shrinking models to fit). Without running models locally without internet, you cannot use an LLM offline, cannot run a voice assistant on a smart speaker without cloud calls, and cannot build a system that survives a network outage.


## The intuition

Think of a model like a well-trained chef. The chef knows many recipes (the stored knowledge or parameters within the model). To serve a dish, the chef needs ingredients (the input) and a kitchen (the hardware). Now imagine two situations. First: the chef lives in a remote village with no market deliveries — they must cook only with what they can grow locally, and they cannot call a mentor for advice. That is edge inference: the model lives on the device (phone, sensor, car) and works with only that device's storage space (memory) and processing power or energy consumption (compute). Note that memory is what the device holds, while compute is how fast it works. Second: the chef works in a kitchen inside a high-security government building where no one can enter or leave, and no phone calls are allowed. That is air-gapped inference: the model and the data stay inside a physically isolated network, never touching the internet. In both cases, the chef cannot fetch missing ingredients from a distant warehouse (no cloud API calls). They must work with what they have. The constraints are not optional — they are the whole point. For edge, it is about speed, privacy, and reliability (no network delay, or latency). For air-gapped, it is about security and legal compliance (data cannot leave). The key ideas from your library — quantisation (shrinking the model) and distillation (teaching a small model to mimic a big one) — are exactly the tools that make these situations possible, because storage space is tight and updates are hard.


## An analogy

Imagine you are a librarian in a small village library. The library has a tiny building (device memory) and no phone line to the big city library (no internet). A villager asks for a recipe for a complex biryani. You have two options: you can walk to the big city every time (that would be a cloud API call — slow, and maybe not allowed if the village is in a locked compound). Or you can keep a small, well-chosen set of books inside your tiny library. That small set is your quantised model — it covers the most common requests, but it might not have every exotic recipe. If someone asks for a recipe not in your small collection, you cannot fetch it; you have to improvise or say no. In an air-gapped setting, the library is inside a military base; no one can bring in new books without a security check, and updates are done by physically carrying a USB stick through a guarded gate. That is the update path: slow, deliberate, and planned in advance. The analogy breaks down where a model is not a library but a compressed memory — the small collection is not a subset of the big city's books but a paraphrased summary, and some knowledge is inevitably lost or distorted, which is why quantisation has a quality cost. Also, unlike a librarian who can reason, the model is just arithmetic; it cannot improvise a truly novel recipe.


## Definition

Edge and air-gapped inference is the practice of running a machine learning model entirely on a local device or inside an isolated network with no internet connection, which forces the model to be small enough to fit in local storage space and designed so that its stored parameters can be updated only through a carefully planned manual or scheduled process.


## How it connects

You have not yet mastered any prerequisite concepts, but your library notes on Inference Optimization mention that this topic belongs there. It connects forward to Local Model Deployment (running models on your own hardware) — edge is a special case where that hardware is a small device. It also relies heavily on Quantization (shrinking parameters to save storage space) and Knowledge Distillation (training a small student to imitate a large teacher) — without those compression tricks, almost no modern model would fit on an edge device. Finally, it touches Network Delay and Speedups, because edge models often trade a little quality for much lower response time. In the book The Craft of Post-Training, this is covered under the post-training stack, which refers to the final steps taken after a model is trained, because quantisation and distillation are typically applied then.


## The mathematics

The mathematics behind deciding if a model fits on an air-gapped device is simple arithmetic: counting, multiplying, and comparing sizes. There is no calculus, no derivatives, and no complex algebra here. You only need to know how much memory the model occupies and whether that amount is smaller than the memory available on the device.

A machine learning model is made of numbers called weights. Each weight is a single number stored in the computer's memory. The total space the model takes up depends on two things:
1. How many weights there are.
2. How much space each individual weight uses.

The space used by one weight is determined by its precision. Precision refers to how detailed the number is. A whole number (like 5) needs less space than a decimal with many places (like 3.14159). In computing, we measure this space in bytes. A common precision for models is called "32-bit floating-point." This means each weight takes up 4 bytes of memory. Another common precision is "8-bit," which takes only 1 byte per weight but is less precise.

To find the total memory the model needs, you multiply the count of weights by the size of one weight. Then, you compare that total to the free memory on your device. If the model's size is less than or equal to the device's memory, it fits. If it is larger, it does not fit, and you must reduce the size (for example, by using lower precision weights).

The formula is:
Total Memory in Bytes = Number of Weights × Bytes per Weight

After calculating the total bytes, you often convert to Kilobytes (KB) because device memory is usually listed in KB or Megabytes (MB). There are 1024 bytes in 1 kilobyte. So, you divide the total bytes by 1024 to get the size in KB.

The decision rule is:
If Total Memory in KB ≤ Device Free Memory in KB, then the model fits.
Otherwise, it does not fit.


## A worked example

Let us walk through a specific case with real numbers to see how this works.

Suppose you have a small model. You count the weights and find there are exactly 1,000 weights in total.
You decide to store these weights using 32-bit floating-point precision. As stated above, each 32-bit weight takes up 4 bytes of memory.
The device you want to run this model on has 4 KB (kilobytes) of free memory available.

Step 1: Calculate the total size in bytes.
You multiply the number of weights by the bytes per weight.
1,000 weights × 4 bytes/weight = 4,000 bytes.
The model occupies 4,000 bytes of memory.

Step 2: Convert the size to kilobytes (KB).
Device memory is usually measured in KB. To convert bytes to KB, you divide by 1024 (because 1 KB = 1024 bytes).
4,000 bytes ÷ 1024 bytes/KB = 3.90625 KB.
For simplicity, we can round this to 3.9 KB.
The model occupies approximately 3.9 KB of memory.

Step 3: Compare the model size to the device memory.
The device has 4 KB of free memory.
The model needs 3.9 KB.
We check if 3.9 is less than or equal to 4.
3.9 ≤ 4 is true.

Result: The model fits on the device. You can deploy it without compression.

What if the model were larger? Suppose the model had 1,500 weights instead of 1,000.
Step 1: 1,500 weights × 4 bytes/weight = 6,000 bytes.
Step 2: 6,000 bytes ÷ 1024 = 5.86 KB (approximately 5.9 KB).
Step 3: Compare 5.9 KB to the device's 4 KB.
5.9 ≤ 4 is false.
The model does not fit. You would need to compress it, for example by using 8-bit weights (1 byte per weight) instead of 32-bit weights, to reduce the total size below 4 KB.


## From scratch, in Python

This code checks whether a model fits on a device. We'll represent the model's weights as a list of numbers, and we'll write a function that calculates the sizes.

```python
# We'll build up from basics: a list is an ordered collection of items.
# Here's a list of example weights (just a few, for illustration):
weights_example = [0.5, -1.2, 2.3, 0.8, -0.4]

# Define a function that calculates the memory needed for a list of weights.
# A function is a reusable block of code that takes inputs and returns a result.
def memory_needed(weights, bytes_per_weight):
    # 'len' tells us how many items are in the list
    number_of_weights = len(weights)
    # Multiply the count by the bytes per weight
    total_bytes = number_of_weights * bytes_per_weight
    # Return the result
    return total_bytes

# Let's test it with the example list and 4 bytes per weight (32-bit float)
size_bytes = memory_needed(weights_example, 4)
print("Size in bytes:", size_bytes)  # Should print 20

# Convert to kilobytes: there are 1024 bytes in a KB
size_kb = size_bytes / 1024
print("Size in KB:", size_kb)  # Should print 0.0195...

# Assume device memory in KB (e.g., 0.02 KB is about 20 bytes)
device_memory_kb = 0.02
if size_kb <= device_memory_kb:
    print("Model fits on device.")
else:
    print("Model does not fit. Need to compress.")
```


## The same thing, with a framework

In practice, you wouldn't hand-roll this; you'd use a library like PyTorch or TensorFlow to inspect model size directly. For example, in PyTorch, `model.num_parameters()` gives the count, and you might use a helper to compute bytes. But underneath, it's the same multiplication. A real deployment check might look like:

```python
# (Pseudo-code, not fully runnable without installing torch)
import torch

model = torch.nn.Linear(10, 5)  # just a tiny example
param_count = sum(p.numel() for p in model.parameters())
bytes_per_param = 4  # if float32
size_bytes = param_count * bytes_per_param
print(size_bytes)
```

This is exactly the same arithmetic: count the weights, multiply by bytes per weight.


## What the code does

The code first defines a list `weights_example` — a list is a container that holds items in order, like a series of numbered boxes. Each weight is a number. Then we define a function `memory_needed` that takes the list and a `bytes_per_weight` value, uses `len` to count how many weights there are, multiplies to get total bytes, and returns it. We call the function with `4` for 32-bit precision. Then we convert bytes to kilobytes by dividing by 1024, just like the numerical example. Finally, we use a conditional `if` to compare the size to the device's memory. If the model is too big, the print statement tells us to compress. This is the core of edge deployment decisions.


## Common misconceptions

**Edge means a Raspberry Pi** — Edge inference only runs on tiny microcontrollers or Raspberry Pi devices.

Edge refers to anywhere *near the user* and away from the cloud — including your laptop, phone, or a server inside a hospital. It is about topology, not size. In fact: Edge is any device that runs the model locally without calling a remote server. It can be as small as a smart sensor or as large as an on-premises workstation.

**Air-gapped means no model updates ever** — Once a model is deployed in an air-gapped environment, it can never be changed.

Updates can happen through planned, manual transfers — like USB sticks or signed media — but they are not routine and require human approval. In fact: Updates are possible but must be designed in advance. The update path is slow and deliberate, not automatic.

**Quantisation always destroys accuracy** — Shrinking a model to fit on edge always ruins its performance.

Modern quantisation techniques (4-bit, 8-bit) often lose only a small amount of accuracy, especially if combined with calibration or quantisation-aware training. In fact: Quantisation trades a small, measurable quality loss for a large gain in memory and speed. In many cases the loss is negligible.

**Edge models are just the same as cloud models but smaller** — You can take any large model and just run it on a phone; the cloud version and edge version behave identically.

Edge models are often architecturally different (fewer layers, distillations) and have different runtime constraints. They are not merely a scaled-down copy. In fact: Edge models are usually trained or compressed specifically for the device, and they may have different capabilities than the full cloud model.

**No internet means no hardware requirements** — If the model doesn't need the internet, it can run on any device.

Even offline, the device still needs enough memory, compute, and power to run the model. A weak device cannot run a large model. In fact: Edge inference still requires matching model size to the device's resources. You cannot ignore compute and memory just because there's no network.


## In short

Edge and air-gapped inference means running models locally with no internet. The model must fit in the device's memory, which usually requires compression (quantisation, distillation). Update paths must be planned because they are manual and slow. It is used everywhere: on phones, laptops, camera sensors, hospitals, and factories, wherever data cannot leave the premises or the network is unreliable. The core arithmetic is: number of weights × bytes per weight = model size, which must fit in available memory.


## Where this is used

Edge and air-gapped inference is everywhere you cannot rely on the internet.

**Large language models**: On-device LLMs, like those running on flagship smartphones, are quantised to 4-bit or 8-bit precision to fit into a few gigabytes of RAM. The developer integrates a runtime like Llama.cpp or TensorFlow Lite, which loads the quantised weights and executes the transformer forward pass locally. When you type a prompt, the entire computation happens on your phone's neural processing unit (NPU) — no data is sent to a server. This is what makes offline assistants or private chats possible. Without this, a 70-billion-parameter model could never run on your device — it would need terabytes of RAM.

**Small language models (on phones/laptops)**: These are models with fewer parameters (e.g., 1–3 billion), often trained via knowledge distillation from a larger teacher. The developer picks a model in the ONNX or GGUF format, quantises it further with a tool like `llama.cpp`'s quantisation, and serves it with a local inference server such as Ollama. The constraint is the device's memory and the available CPU/GPU. The same software runs on a laptop for offline coding assistance or on a smart speaker for wake-word detection.

**Retrieval-augmented generation (RAG)**: A key challenge in edge RAG is that the embedding model and the retriever must also run locally. The developer embeds a local vector database (like SQLite with a vector extension) and a small embedding model (e.g., a quantised MiniLM) to index documents on the device. When a user queries, the embedding of the query is computed locally, and the retriever searches the index — entirely on-device. In an air-gapped enterprise setting, this allows RAG over confidential documents without any external API call.

**Agentic systems**: Agents that plan, call tools, and act in loops — when deployed at the edge — must be self-contained. The developer writes a loop that calls the local model, which emits a tool call (like a JSON action) instead of a network request. Tools are local functions (file I/O, sensors) rather than web APIs. In an air-gapped environment, the agent cannot phone home for new instructions, so the system must be fully deterministic from local weights. This is common in industrial automation where robots use on-device vision models to decide actions without cloud round-trips.

**Everyday applied ML**: Recommendation systems on your streaming app may run a small on-device model to personalise offline (e.g., Netflix's on-device ranking). Vision models on security cameras detect people without sending video to the cloud — preserving bandwidth and privacy. Fraud detection in banking often runs on-premises in an air-gapped network because transaction data is sensitive. Speech recognition in a smart speaker's wake word engine runs entirely on the chip to avoid sending audio to the server. In all these, the developer's workflow is: train the model in the cloud, compress it (quantisation, distillation), then export it to a mobile/edge format (TFLite, Core ML, ONNX). What breaks when done badly: if the model is too large, it doesn't load and the app crashes; if quantisation is too aggressive, accuracy degrades; if updates are not planned, you cannot improve the model without expensive manual intervention.

In summary, this concept is foundational to any deployment where the cloud is not an option.


---

Read this on the web: https://basavapurushottam.com/learn-ai/edge-and-air-gapped-inference
