# Model Extraction

Reconstructing a proprietary model's behaviour by querying it systematically and training a substitute on the responses.

**Rests on:** Knowledge Distillation

*Security and Privacy · AI Security*


## Why learn this

Model extraction matters because it allows someone to copy the capabilities of a valuable, private system without paying for access or having permission. Imagine a company has spent years training an AI on their confidential documents to answer customer questions. If an outsider can replicate that AI’s behavior exactly, they now have a free tool that competes with the company’s product, using the company’s hard-won knowledge. This undermines the business model of any service that sells specialized AI access. Understanding this threat is essential for anyone who designs systems where proprietary intelligence is the core asset, because it shows that security cannot rely solely on hiding source code; you must also protect the system from being studied through its outputs.


## The intuition

Imagine you are trying to learn a chef’s secret recipe, but you are not allowed into their kitchen. You cannot see their ingredients or watch them cook. However, you can sit at their counter and order dishes. The chef brings out dish number one: it tastes like chicken with a hint of lemon and garlic. You write that down. Dish number two tastes like fish with the same lemon-garlic note. Dish number three is lamb with that same profile. After 100 such meals, you realize the chef’s unique signature is always this specific lemon-garlic blend, even if the main ingredient changes. You have now reconstructed the 'essence' of their cooking style without ever seeing their notebook. 

In AI terms, the 'chef' is the proprietary model. The 'notebook' contains its weights—the internal numbers that define how it processes information. You cannot access these weights directly. Instead, you interact with the model through an API (Application Programming Interface), which is a digital gateway that lets you send it data and receive answers. This API is often protected by rate limiting, which is a rule that prevents anyone from sending too many requests in a short time—for example, allowing only one question every 10 seconds. 

A model extractor works by sending thousands of carefully chosen questions to the API and recording the exact outputs. They then train their own system—a substitute model—on these Q&A pairs. The substitute model learns to mimic the original’s responses so closely that if you asked both models the same question, they would give identical answers. The extractor now has a working copy of the proprietary intelligence, derived entirely from observing its behavior rather than stealing its code.


## An analogy

Think of the original model as a locked digital black box with only a single input slot and a single output screen. You can slide paper in (your question) and read the typed answer on the screen, but you cannot open the box to look at the gears inside. Those internal gears are called weights, which are just billions of numbers adjusting how data moves through the system. To prevent people from sliding papers in too fast, there is a guard called rate limiting who stops you if you try to send more than five inputs per minute. 

Model extraction is like an artist standing before a locked glass case containing a rare painting. They cannot touch the painting. But they can look at it for hours every day, sketching what they see. Over time, their sketch becomes so detailed that anyone looking at it would think it is the original. The artist has not stolen the painting; they have stolen its *appearance* by observing its outputs. This works because the model’s answers (outputs) are a direct reflection of its internal weights. By mapping enough inputs to outputs, you reverse-engineer the relationship, building your own 'sketch' or substitute model that behaves identically to the locked box.


## Definition

Model extraction is the act of reconstructing a proprietary AI system's decision-making process by querying it repeatedly and training a separate, public model to mimic its responses, thereby creating a functional copy without accessing the original's private parameters.


## How it connects

Model extraction sits alongside adversarial examples, which are modified inputs designed to trick a model into making errors; while extraction copies behavior, adversarial examples exploit weaknesses in that behavior. It is also closely related to data poisoning, where attackers corrupt the training data used to build models, whereas extraction targets trained, deployed models directly. Understanding these helps distinguish between attacking how a model learns (poisoning) versus stealing what it has already learned (extraction).


## The mathematics

The mathematics of model extraction is not advanced calculus — it is arithmetic and a bit of geometry. The core idea is that a model, whatever it is, processes inputs and produces outputs. If you send in a number and get a number out, you can record that pair. Do that for many inputs, and you have a table of examples. Now you want to build your own model that matches that table. That involves training a new model on the recorded pairs. Training itself involves a concept called a loss function, which measures how wrong the model's predictions are. You adjust the model's parameters to make the loss small. This is where a tiny bit of calculus appears, but we will only scratch the surface. The key mathematical operations here are: mapping inputs to outputs (a function), measuring the difference between two outputs (a number you can calculate), and adjusting numbers to reduce that difference (a process you can do by hand for small examples). None of this is beyond arithmetic. The subtlety is that the model is a high-dimensional function — it takes millions of inputs and produces thousands of outputs — but the principle is the same as for a single number.

To make this precise, we need to define our terms clearly. Let f be the original model. In mathematical language, a model is a function. A function is a rule that takes an input and gives exactly one output. We call the substitute model g. We also call it a predictor because its job is to predict the output of f.

We do not look inside f. Instead, we treat it as a black box. We choose a set of inputs. Let us call this set x₁, x₂, ..., xₙ. Here, n is just a number telling us how many inputs we are using. For example, if we use 100 inputs, then n = 100.

For each input xᵢ (read as 'x sub i'), we send it to the original model f. The model returns an output. We call this output yᵢ. So, y₁ is the output for input x₁, y₂ is the output for input x₂, and so on. We now have a list of pairs: (x₁, y₁), (x₂, y₂), ..., (xₙ, yₙ). This list is our training data.

Our goal is to find a function g that produces outputs close to yᵢ when we give it the input xᵢ. To measure how close g is, we use a number called the Mean Squared Error, or MSE. The word 'mean' means average. The word 'squared' means we multiply the difference by itself.

Here is how we calculate MSE step by step:
1. For each pair i from 1 to n, take the output of our substitute model g(xᵢ).
2. Subtract the true output yᵢ from this value: (g(xᵢ) - yᵢ). This is the error for that single point.
3. Square this error: (g(xᵢ) - yᵢ)². We square it so that positive and negative errors do not cancel each other out, and so that larger errors count more heavily.
4. Add up all these squared errors for all n points.
5. Divide the total sum by n. This gives the average squared error.

The formula is:
MSE = (1/n) * [ (g(x₁) - y₁)² + (g(x₂) - y₂)² + ... + (g(xₙ) - yₙ)² ]

We want to choose the parameters of g so that this MSE number is as small as possible. This process of finding the best parameters is called training. The substitute model g is the extracted model.


## A worked example

```text
Suppose the original model is a black box that takes a single number x and outputs a single number y. We want to extract a substitute model that behaves similarly, at least for the range of x we care about. The original model is not a simple formula; it is a complex function, but we can query it. We decide to ask for three points: x=1 gives y=2, x=2 gives y=4, x=3 gives y=6.

Step 1: Query 1: x=1, answer y=2   # We send 1 to the API and get back 2.
Step 2: Query 2: x=2, answer y=4   # Send 2, get 4.
Step 3: Query 3: x=3, answer y=6   # Send 3, get 6.
Step 4: Now we fit a simple line: y = mx + b   # We assume the substitute is a straight line with slope m and intercept b.
Step 5: Using point (1,2): 2 = m(1) + b   # Substitute the first pair.
Step 6: Using point (2,4): 4 = m(2) + b   # Substitute the second pair.
Step 7: Subtract the first equation from the second: (4 - 2) = m(2 - 1) → 2 = m   # We eliminate b and find m.
Step 8: Then from 2 = 2(1) + b → b = 0   # Plug m back to find b.
Step 9: Substitute model: y = 2x   # We have reconstructed a simple model from three queries.

Result: The substitute model is y = 2x, which perfectly matches the three queried points. With more queries, we could approximate a more complex original model. In practice, the original model is not a line, so we need many more points and a more flexible substitute (like a neural network).
```


## From scratch, in Python

```python
# We will simulate a black-box model that we want to extract.
# In real life, this would be an API call. Here, it's a simple function.
def original_model(x):
    # Pretend this is a proprietary model. We cannot see inside.
    return 2 * x + 1

# Our substitute model: a linear function with learnable parameters m and b.
# We start with random guesses.
m = 0.0
b = 0.0

# We query the original model at a few points to build our training data.
# In real extraction, we would send many queries.
x_values = [1, 2, 3, 4]
y_values = [original_model(x) for x in x_values]
print('Queries:')
for x, y in zip(x_values, y_values):
    print(f'  x={x} → y={y}')

# Now we train our substitute by trying to match these points.
# We use a simple method: adjust m and b to reduce the error.
# We repeat many times (epochs).
learning_rate = 0.01
for epoch in range(100):
    total_error = 0.0
    # For each input-output pair, compute prediction and error.
    for x, true_y in zip(x_values, y_values):
        pred = m * x + b
        error = pred - true_y
        total_error += error ** 2
        # Update m and b to reduce error: gradient descent (simplified).
        # We move m and b a little in the direction that reduces the error.
        m = m - learning_rate * 2 * error * x
        b = b - learning_rate * 2 * error
    if epoch % 20 == 0:
        print(f'Epoch {epoch}: mean squared error = {total_error / len(x_values):.4f}')

print(f'Final substitute: y = {m:.2f}x + {b:.2f}')
print('Compare to original: y = 2x + 1')
```


## What the code does

First, we define the original model as a simple linear function. In real life, this would be an API call you cannot see inside. Then we set up our substitute model with two parameters, m and b, both starting at 0. Next, we query the original model at four points to build a small training set. We print those queries. Then we loop 100 times over the data. For each pair, we compute the prediction from our substitute, the error (difference), and update m and b using a simple rule that reduces the error. This is a simplified gradient descent. After 100 epochs, we print the learned m and b, which should be close to the original 2 and 1. The key idea: we only ever see the original model's outputs, never its internal parameters, yet we can reproduce its behaviour.


## Common misconceptions

**Extraction is the same as hacking into the model's servers.** — To steal a model, you must break into its infrastructure and copy the weights.

Extraction works purely through the public interface. You never touch the model's internal weights or servers. You only send inputs and receive outputs, just like a normal user. In fact: Extraction is a black-box attack: it reconstructs behaviour from query responses, no intrusion required.

**The extracted model is always a perfect copy.** — If I query enough, I can get the exact same weights as the original.

The substitute learns the input-output mapping, not the internal parameters. Many different sets of weights can produce the same behaviour. Also, you only sample a finite set of inputs, so the substitute may behave differently on unseen inputs. In fact: The substitute approximates the original's behaviour, not its parameters. Accuracy depends on query coverage and the substitute's capacity.

**Rate limiting is a complete defence.** — If we limit how many queries a user can make, extraction is impossible.

Rate limiting slows extraction but does not stop it. An attacker can distribute queries over time, use multiple accounts, or focus on a critical subset of inputs. It raises cost, not capability. In fact: Rate limiting is a practical deterrent that increases the attacker's cost, but a determined adversary can work around it. Output granularity (e.g., hiding confidence scores) is a stronger defence.

**Extraction is only about stealing the model's weights.** — The point of extraction is to get the weights so I can use the model for free.

The value is often in the behaviour, not the weights. A substitute can be used for offline attack development (training adversarial examples), for probing vulnerabilities, or for competitive intelligence. In fact: Extraction steals capability and enables further attacks, even if the weights are never recovered.


## In short

Model extraction is the art of cloning a model's behaviour by querying it and training a substitute on the responses. It works without ever seeing the internal weights. The key defences are rate limiting and controlling what the API reveals (e.g., avoiding detailed confidence scores). Extracted substitutes are useful for offline attacks and reverse engineering. Always assume that any model you expose via an API can be partially extracted — so protect it with throttling and output sanitisation.


## Where this is used

Model extraction is a real threat across the AI ecosystem. In large language models (LLMs), an attacker can query a commercial API (like GPT-4 or Claude) thousands of times, collecting prompt-response pairs. They then use those pairs to fine-tune an open-source model (like Llama) to mimic the proprietary model's behaviour. This is exactly how some open-source 'reproductions' have been built. The extracted model is often smaller and cheaper to run, but it may inherit the original's capabilities — including potential weaknesses. In small models running on devices (like smart speakers or phone keyboards), extraction can happen by sending many voice commands or text inputs and using the responses to train a local substitute. The substitute may be used offline to develop attacks, as your library notes. In retrieval-augmented generation (RAG), extraction can target the retrieval component: by querying with carefully crafted prompts, an attacker can infer what documents are in the knowledge base and even extract the embeddings, allowing them to reconstruct the private data. In agentic systems that plan and call tools, extraction can go further: because the agent's outputs reveal its decision-making process (tool calls, intermediate reasoning), an attacker can learn not just the model's weights but its planning rules — and then build a substitute agent that mimics the tool usage. In everyday applied ML (recommendation, fraud detection, speech recognition), extraction is common: an attacker sends crafted inputs (e.g., user profiles to a recommender) and observes the scores, then trains a shadow model to predict those scores. This shadow model can be used to manipulate the original system (e.g., finding items that get high ratings unfairly). What does a developer actually write? They write a query loop that sends a batch of inputs, records outputs, and uses them to train a substitute. They might use a library like `openai` or `requests` to hit the API, then use `torch` or `sklearn` to train the substitute. What breaks when done badly? If the attacker does not select queries carefully, the substitute might be a poor approximation — for example, only querying in a narrow range. Defences: rate limiting (limiting how many queries a user can make in a short time) and output granularity (e.g., returning only the top class and not confidence scores, or rounding logits). Also, monitoring for unusual query patterns. In practice, extraction is a cat-and-mouse game: attackers diversify query patterns, defenders look for anomalies.


---

Read this on the web: https://basavapurushottam.com/learn-ai/model-extraction
