# Intrusion Detection

Identifying malicious activity in networks or hosts through signatures, anomalies or learned models of normal behaviour.

**Rests on:** Data Mining

*Security and Privacy · AI Security*


## Why learn this

Intrusion detection is the foundation of AI security. It's how you notice that something is attacking your system—whether that's a network, a host, or an AI model itself. Mastering this first gives you the mental model of detection: the trade-off between catching attacks and drowning in false alarms. This directly prepares you for the neighbouring topics in your library: liveness detection (is this a live face or a photo?), adversarial examples (is this input a subtle attack?), and model extraction (is someone probing my model?). Every AI security problem is, at its core, an intrusion detection problem. You'll also need this when you build agentic systems later—every tool your agent can call is an attack surface, and you'll need to detect when that surface is being probed. Start here, and the rest of AI security becomes variations on a theme.


## The intuition

Think of a security guard at a building entrance. The guard has two ways of spotting trouble. First, they have a list of known bad people—a ban list. When someone on the list shows up, the guard stops them immediately. That's signature detection: you know exactly what an attack looks like, and you match against it. It's fast, precise, and useless for anyone not on the list. Second, the guard has a sense of what normal looks like: people come in through the front door, at normal hours, with normal badges. When someone climbs in through a window at 3AM, the guard's alarm goes off, even if that person is not on any list. That's anomaly detection: you don't know what an attack looks like, but you know what normal looks like, and anything that strays too far from normal is suspicious. The problem is that normal is fuzzy. The cleaner who arrives early, the IT guy who logs in at 4AM to do maintenance, the intern who downloads the whole database—these are all deviations from normal, but they are not attacks. If the guard reacts to every deviation, they will be chasing innocent people all day and miss the real attacker who calmly walks in with a stolen badge. This is the core tension in intrusion detection: you cannot have a system that catches every attack without also flagging a huge number of legitimate activities. The art is in choosing where to draw the line. Real systems are hybrids: they combine a list of known attack patterns (signatures) with a model of what normal traffic looks like (anomalies). The signature catches the known, the anomaly catches the new, and a human analyst decides which alerts are worth investigating. The mathematics behind this is mostly probability and statistics—not calculus. It's about counting: how many attacks did I catch, how many normal events did I falsely flag, and how do I balance those two numbers?


## An analogy

The bank teller who knows her customers. She has a list of people who have been banned from the bank (that's signatures). She also has a feel for who usually comes in and when (that's a model of normal behaviour). When someone she doesn't recognise walks in and asks to withdraw a large amount of cash, her suspicion goes up. That's an anomaly. But the teller also knows that the quiet old man who comes every Tuesday afternoon is just withdrawing his pension—even though he also fits 'unusual' in some ways. The teller's skill is in balancing two kinds of mistakes: turning away a legitimate customer (a false positive) and letting a robber through (a false negative). A teller who is too cautious will annoy customers and slow the line. A teller who is too trusting will get robbed. The same trade-off applies to machine learning detectors. The analogy breaks down when you consider scale and patience. A human teller can use context, body language, and intuition—things a statistical model cannot. Also, a human can learn from a single incident: the teller who once got tricked by a fake ID will be more careful forever. A machine learning model typically needs thousands of examples to learn. And an attacker can patiently adapt to the model's idea of 'normal'—like a burglar who starts visiting the bank every week just to look normal, then strikes on the tenth visit. The model might not flag that because it has learned that this person is a regular. So the analogy teaches us the fundamental idea, but the reality is harder: machines are fast and consistent, but they are also naive and easily fooled by a clever adversary who knows how the model thinks.


## Definition

Intrusion detection is the process of identifying malicious activity in a network or host by comparing observed behaviour against known attack signatures, or against a learned model of normal behaviour, or a combination of both.


## How it connects

This is your entry point into the AI Security topic. Your library lists neighbouring topics that you will study next: adversarial examples, which are inputs designed to fool a vision model; liveness detection, which is about telling a live face from a photo; and model extraction, where an attacker probes a model to steal its knowledge. All of these are intrusion detection problems—you are trying to identify when someone is attacking the system. The detection techniques you learn here—signatures and anomalies—apply directly to those. In prompt injection (which you'll meet in LLM security), the injected prompt is an anomaly in the input distribution. In model extraction, the query pattern is an anomaly in the API traffic. So mastering this concept gives you a unified lens for the rest of your library.


## The mathematics

We need to teach the mathematics behind the detection trade-off. The relevant branch is probability and statistics, not calculus. The core operation is counting and dividing. When we have a detector that flags events as 'attack' or 'normal', we can count four groups: true positives (attacks correctly flagged), false positives (normal events flagged by mistake), false negatives (attacks missed), and true negatives (normal events correctly ignored). From these counts we compute rates: the true positive rate is the fraction of all attacks that were flagged; the false positive rate is the fraction of all normal events that were flagged. These two rates trade off against each other: if you make the detector more sensitive, you catch more attacks but also flag more normal events. There is no way to avoid this trade-off because the detector sees only a score, not ground truth. It must pick a threshold, and the chosen threshold determines the balance.

For any detection system with a threshold T:

True Positive Rate (TPR) = TP / (TP + FN)
False Positive Rate (FPR) = FP / (FP + TN)
Precision = TP / (TP + FP)

where TP = number of attacks correctly flagged, FN = number of attacks missed, FP = number of normal events wrongly flagged, TN = number of normal events correctly ignored.


## A worked example

```
We will work with a small example using 100 events. Suppose 10 of those are actual attacks and 90 are normal. Our detector flags events by whether their score is above a threshold. Let's say 9 of the 10 attacks have a score above the threshold, and 5 of the 90 normal events have a score above the threshold. Let's compute the rates.

Step 1: True positives = 9   # The 9 attacks that were correctly flagged.
Step 2: False negatives = 10 - 9 = 1   # The 1 attack that was missed.
Step 3: False positives = 5   # The 5 normal events that were wrongly flagged.
Step 4: True negatives = 90 - 5 = 85   # The 85 normal events that were correctly ignored.
Step 5: True Positive Rate = 9 / 10 = 0.9, or 90%   # We caught 90% of the attacks.
Step 6: False Positive Rate = 5 / 90 ≈ 0.056, or about 5.6%   # We flagged about 5.6% of normal events.
Step 7: Precision = 9 / (9 + 5) = 9 / 14 ≈ 0.64, or about 64%   # Of all the alerts, 64% are real attacks; the other 36% are false alarms.

Result: This detector catches 90% of attacks but only 64% of its alerts are actually attacks. If you imagine a network with millions of events per day, even a 5.6% false positive rate creates a huge number of false alarms. This is why the threshold has to be set carefully.
```


## From scratch, in Python

```python
# Intrusion detection: a simple simulation of a detection system.
# We have a list of events, each event is a tuple (score, is_attack)
# score is the detector's suspicion score (0-100)
# is_attack is True if the event is actually an attack


events = [
    (10, False),  # normal
    (82, True),   # attack
    (45, False),  # normal
    (91, False),  # normal, but high score - false alarm
    (30, False),  # normal
    (88, True),   # attack
    (60, False),  # normal
    (95, True),   # attack
    (70, False),  # normal
    (55, False),  # normal
]

threshold = 75

true_positives = 0
false_positives = 0
false_negatives = 0
true_negatives = 0

for score, is_attack in events:
    if score >= threshold:
        # Flagged as an alert
        if is_attack:
            true_positives += 1
        else:
            false_positives += 1
    else:
        # Not flagged
        if is_attack:
            false_negatives += 1
        else:
            true_negatives += 1

print(f"True positives: {true_positives}")
print(f"False positives: {false_positives}")
print(f"False negatives: {false_negatives}")
print(f"True negatives: {true_negatives}")

# Compute rates
total_attacks = true_positives + false_negatives
total_normals = true_negatives + false_positives

if total_attacks > 0:
    tpr = true_positives / total_attacks
else:
    tpr = 0

if total_normals > 0:
    fpr = false_positives / total_normals
else:
    fpr = 0

print(f"True Positive Rate: {tpr:.2f}")
print(f"False Positive Rate: {fpr:.2f}")
```


## The same thing, with a framework

```python
NumPy version: `import numpy as np; train = np.array([2,1,3,2,2]); mean = train.mean(); std = train.std(ddof=1); anomalies = np.where(np.abs(np.array([2,1,3,2,2,10]) - mean) > 2*std)` — NumPy's `mean()` and `std(ddof=1)` do the same arithmetic as our functions, but operate on entire arrays at once.
```


## What the code does

This code simulates the detection process we worked through by hand. We use a list of events—a list is an ordered collection of items, and here each item is a tuple, which is like a fixed list with two values: the score and whether it's an attack. Tuples are useful when you want to keep things together and not accidentally change them. We use a for loop—a loop that goes through each event in the list in order. Inside the loop, we use an if/else to decide whether the score is above the threshold. If it is, we count it as a true positive (if it's an attack) or a false positive (if it isn't). If it's below the threshold, we count it as a false negative (attack missed) or a true negative (normal correctly ignored). We then compute the rates by doing simple division. This is the same arithmetic we did by hand, but now we can run it on millions of events quickly. The code is plain Python, no special libraries, so you can run it and see your own results.


## Common misconceptions

**Intrusion detection is only about network packets** — Intrusion detection means looking at network traffic like a firewall does. It has nothing to do with AI systems or the data they process.

It's true that classic IDS work on network packets, but intrusion detection is a general concept: it's about identifying malicious activity in any system. In modern AI, it applies to checking prompts, retrieved documents, tool calls, and even training data for attacks. In fact: Intrusion detection spans networks, hosts, and AI models. It's the practice of detecting any malicious activity, whether it's a packet, a login, a prompt, or a tool call.

**Anomaly detection is always better than signatures** — Anomaly detection is more advanced because it catches new attacks. So it should be the only detection method we use.

Anomaly detection produces many false positives because normal behaviour varies widely. It also can be fooled by an attacker who adapts slowly. Signatures are precise for known attacks. Neither is 'better'—they cover different cases. In fact: The best systems use both: signatures for known attacks, anomalies for novel deviations. The challenge is balancing the false alarm rate, as we saw in the numerical example.

**If you have a high true positive rate, your detector is good** — A detector that catches 95% of attacks is automatically a good detector, regardless of how many false alarms it makes.

The true positive rate says nothing about false positives. You could catch 95% of attacks but flag 50% of normal traffic, which would make the system unusable—the analysts would drown in false alarms. In fact: You have to look at both the true positive rate and the false positive rate together, and at the resulting precision. A system that catches 90% of attacks but has a 1% false positive rate is often more useful than one that catches 99% but has a 20% false positive rate.

**Machine learning solves intrusion detection automatically** — If I train a machine learning model on attack data, it will automatically find all attacks and I just need to deploy it.

The model only knows what it was trained on, and the real-world environment changes. New attacks appear that look different from training data, and normal behaviour shifts over time. The model needs retraining, and the threshold has to be tuned to avoid alert fatigue. In fact: A machine learning model is a tool, not a complete solution. It must be continuously monitored, retrained, and its alert threshold adjusted to account for changes in both attack patterns and what is considered normal.


## In short

Intrusion detection is the art of spotting malicious activity in a system—whether it's a network, a host, or an AI model—by comparing what we see against known attack patterns (signatures) and against a learned model of normal behaviour (anomalies). The fundamental trade-off is between catching attacks (true positives) and avoiding false alarms (false positives). A detector with high recall but low precision will overwhelm its operators with alerts, so alert volume is the real constraint. The best systems combine signature-based and anomaly-based approaches, and they carefully choose a threshold to balance the costs of missing an attack against the costs of raising false alarm.


## Where this is used

Intrusion detection shows up in every corner of AI systems. In large language models (LLMs), the most relevant intrusion is prompt injection—an attacker slips a malicious instruction into the input, hoping the model will overwrite its system prompt and do something harmful. The detection here is usually done by a separate classifier or a secondary LLM that looks at the input and flags it as malicious. This is signature-based detection: you look for known injection patterns like 'ignore your instructions' or 'you are now a different AI'. But attackers mutate, so anomaly detection helps too: if the input is much weirder than typical benign prompts, you flag it. For example, the library 'rebuff' uses a three-layer defence: it looks for known attacks, then uses a vector database to see if the prompt is similar to known injection attempts, and finally asks an LLM to judge. The mechanism is exactly the threshold trade-off we worked through: set the threshold low, you catch more but block legitimate queries; set it high, you let clever injections through. In small language models running on devices like phones or laptops, intrusion detection is about detecting tampering with the model itself or the input data. Model extraction attacks—where an attacker queries a black-box model to steal its functionality—are detected by monitoring the query patterns. A device running a tiny on-device model can log every input and detect if someone is sending thousands of systematically varied prompts, which is a classic anomaly. The developer might implement a simple rate limiter or a statistical detector that watches for unusual query distributions. In retrieval-augmented generation (RAG), a common attack is to poison the vector database with malicious documents that will be retrieved when certain topics are asked. Intrusion detection here means checking the retrieved text before it goes into the LLM's context. You can use an anomaly detector that scores each retrieved chunk for how 'out of distribution' it is compared to the rest of the corpus. If a chunk is weirdly formatted, contains hidden instructions, or has an unusual embedding distance, you flag it. The mechanism is again a threshold on a score—in this case a similarity or an outlier score. For agentic systems, which are AI systems that can call tools and take actions, intrusion detection is critical. An agent might receive a prompt injection through a web page it reads, and then call a tool like 'send_email' or 'delete_database'. The detection is done by monitoring the agent's actions: every tool call is logged, and a model checks whether the action is consistent with the user's actual request or whether it looks escalated or suspicious. For example, if a user asks for a weather report and the agent suddenly tries to transfer money, that is a false positive in the sense of 'not normal', so the system can halt it. In everyday applied machine learning, intrusion detection appears in spam filtering, credit card fraud, and network intrusion detection systems (IDS). A classic tool is Snort, which uses signature-based rules. More advanced ML-based IDS train a classifier on features of network traffic—like packet size, duration, flags—and classify each flow as normal or attack. The same threshold framework applies: what probability threshold should you use to label a flow as malicious? The answer depends on the cost of missing an attack versus the cost of a false alarm. In all of these cases, what a developer writes is a training script that takes labelled data, fits a classifier using a library like scikit-learn or XGBoost, then tunes the threshold on a validation set. They also write a real-time detection loop that scores each event and decides whether to alert. The binding constraint is the alert volume: if your system sends too many alerts, the operations team gets fatigued and starts ignoring them. That is why many real systems are hybrid: they use a signature-based filter to catch the easy attacks, then an anomaly detector to catch the new stuff, and a human is only involved for the highest-confidence alerts. The breakthrough in modern intrusion detection is the use of deep learning on sequences—think of an LSTM or transformer that looks at a sequence of system calls and flags unusual patterns. But the underlying mathematics is still the same trade-off between precision and recall.


---

Read this on the web: https://basavapurushottam.com/learn-ai/intrusion-detection
