# Data Mining

Discovering patterns in large datasets at the intersection of machine learning, statistics and database systems.

**Rests on:** Big-O Notation, Descriptive Statistics

**Leads to:** Apriori Frequent Itemsets, Entity Resolution, Intrusion Detection

*Machine Learning · Machine Learning Foundations*


## Why learn this

This concept is the doorway to the rest of your Machine Learning Foundations. You will meet it again in every applied topic you study: Decision Trees, K-Nearest Neighbours, Naive Bayes, Linear and Logistic Regression are all data mining algorithms. Churn Prediction, Crop Disease Detection, and even the small models that run on your phone all start with the same question this page answers: how do you find patterns in a pile of data without being told where to look? Once you can spot an association, a cluster, or an outlier, you can build the models that turn those patterns into decisions. This page gives you the vocabulary and the caution you need before you learn any single algorithm, so you do not mistake noise for signal.


## The intuition

Data mining is what you do when you have a huge pile of data and you do not know what is in it. You are not testing a specific idea; you are letting the data speak. Imagine a supermarket that logs every purchase made by every customer for a year. That is millions of rows. No human can read them all. Data mining is the set of techniques that scan that pile and say: 'people who buy nappies also tend to buy beer', or 'there is a group of customers who only shop on weekends and always buy the same brand of milk', or 'this one transaction is so different from every other one that something must have gone wrong'. The key idea is that you did not ask for these patterns in advance. You just said 'find me something interesting'. That sounds easy, but it is not. If you search hard enough in any large pile, you will find patterns that are pure coincidence. Spend a day flipping a coin and you will find a run of ten heads in a row somewhere, and it will mean nothing. The whole skill of data mining is learning to tell a real pattern from a coincidence, and that skill starts here.


## An analogy

Think of a market researcher in a crowded bazaar. They stand in the middle and watch thousands of shoppers walk past. They are not testing a hypothesis; they are just watching. After a while they notice: every time someone buys mangoes, they also buy a newspaper. That is an association. They also notice that the shoppers sort themselves into groups — the early-morning crowd all buy vegetables, the lunchtime crowd all buy ready-made food. That is a clustering. And once in a while they see a shopper buy a hundred packets of salt at once, which is so unusual that it deserves investigation — that is an anomaly. The market researcher is a data miner. Now here is where the analogy breaks down: the researcher is one person with one set of eyes, and unusual patterns stand out to them because of experience. A computer has no experience. It will flag a thousand patterns, and most of them will be meaningless. The researcher can also ask 'why?' — the computer can only report what it sees. So the computer is the tireless scout, and the human is still the one who decides what the pattern means. Data mining is the art of making the scout useful without being drowned by its enthusiasm.


## Definition

Data mining is the process of discovering patterns in large datasets by automatically searching for associations, clusters, and anomalies, using methods from statistics and machine learning, without a predetermined hypothesis.


## How it connects

You have not met any other topics in your library yet, so this page stands alone as your first step. But from here the path is clear: the patterns you find with data mining are exactly what the supervised algorithms in your library — Decision Trees, K-Nearest Neighbours, Logistic Regression — are built to exploit. A decision tree is literally a set of rules for splitting data, which is a way of finding associations. K-Means Clustering, which appears in your notes, is the clustering task in action. And every one of those applied problems — Churn Prediction, Crop Disease Detection — starts with a data mining step to understand what is in the data before building a model. So think of this page as laying the foundation stones: pattern, signal, noise, and the three classic tasks. Every later concept will build on these.


## The mathematics

Data mining does not rely on a single magic formula. Instead, it uses a small set of arithmetic tools to answer three common questions about data: Do two things happen together? Which items are similar to each other? Is one item strange or different from the rest?

Before we look at the numbers, you need to understand that all these methods start with counting and comparing. The core idea is that if a pattern appears often enough in your sample of data, it is likely to be real, not just a coincidence.

1. Association: Finding rules like "If a customer buys bread, they often buy butter." We measure this with two numbers:
   - Support: How common is the pair? It is the count of transactions containing both items divided by the total number of transactions.
   - Confidence: If the first item appears, how likely is the second? It is the count of transactions with both items divided by the count of transactions with just the first item.

2. Clustering: Grouping similar items together. To do this, we need a way to measure "similarity." In data mining, similarity is usually measured by distance. The smaller the distance, the more similar the items. We use a standard method called Euclidean distance. It works by taking the difference between each number in one item and the corresponding number in another item, squaring those differences, adding them up, and then taking the square root.

3. Anomaly Detection: Finding outliers. An anomaly is a data point that is very far away from the group it belongs to. We calculate the distance from a point to the center of its cluster. If this distance is much larger than the typical distance of other points in that same cluster, we flag it as an anomaly.

There is no calculus, no derivatives, and no complex algebra here. Just addition, multiplication, division, squaring, and square roots.


## A worked example

We have a small dataset of six customers. For each customer, we record two numbers: their age and their monthly spending in rupees. Our goal is to see if age and spending are associated, and to group customers into clusters based on similarity.

The data is as follows:
- Customer A: Age 25, Spending ₹2000
- Customer B: Age 30, Spending ₹2500
- Customer C: Age 35, Spending ₹3000
- Customer D: Age 40, Spending ₹4000
- Customer E: Age 45, Spending ₹4500
- Customer F: Age 28, Spending ₹2100

Step 1: Calculate the average (mean) age.
Add all ages together: 25 + 30 + 35 + 40 + 45 + 28 = 203.
Divide by the number of customers (6): 203 / 6 = 33.83.
The average age is approximately 33.83 years.

Step 2: Calculate the average (mean) spending.
Add all spending amounts together: 2000 + 2500 + 3000 + 4000 + 4500 + 2100 = 18100.
Divide by the number of customers (6): 18100 / 6 = 3016.67.
The average spending is approximately ₹3016.67.

Step 3: Check for association between age and spending.
We compare each customer's values to the averages we just calculated. If a customer is older than average AND spends more than average, they are in the "high-high" group. If they are younger than average AND spend less than average, they are in the "low-low" group. These two patterns show that age and spending move together (positive association).

- Customer A: Age 25 is below 33.83. Spending 2000 is below 3016.67. (Low-Low match)
- Customer B: Age 30 is below 33.83. Spending 2500 is below 3016.67. (Low-Low match)
- Customer C: Age 35 is above 33.83. Spending 3000 is below 3016.67. (Mixed: High age, Low spend)
- Customer D: Age 40 is above 33.83. Spending 4000 is above 3016.67. (High-High match)
- Customer E: Age 45 is above 33.83. Spending 4500 is above 3016.67. (High-High match)
- Customer F: Age 28 is below 33.83. Spending 2100 is below 3016.67. (Low-Low match)

Count the matches:
- Low-Low matches: A, B, F (3 customers)
- High-High matches: D, E (2 customers)
- Mixed: C (1 customer)

Five out of six customers follow the pattern that higher age goes with higher spending. This suggests a strong positive association.

Step 4: Calculate distances for clustering.
To group customers, we measure how far apart they are using Euclidean distance. The formula is:
Distance = square root of [ (Age1 - Age2) squared + (Spending1 - Spending2) squared ]

Let's calculate the distance between Customer A and Customer B:
- Difference in age: 25 - 30 = -5. Squared: (-5) * (-5) = 25.
- Difference in spending: 2000 - 2500 = -500. Squared: (-500) * (-500) = 250,000.
- Sum of squares: 25 + 250,000 = 250,025.
- Square root of 250,025 ≈ 500.02.
The distance between A and B is approximately 500.02.

Let's calculate the distance between Customer A and Customer F (to see if they are close):
- Difference in age: 25 - 28 = -3. Squared: 9.
- Difference in spending: 2000 - 2100 = -100. Squared: 10,000.
- Sum of squares: 9 + 10,000 = 10,009.
- Square root of 10,009 ≈ 100.04.
The distance between A and F is approximately 100.04.

Let's calculate the distance between Customer B and Customer F:
- Difference in age: 30 - 28 = 2. Squared: 4.
- Difference in spending: 2500 - 2100 = 400. Squared: 160,000.
- Sum of squares: 4 + 160,000 = 160,004.
- Square root of 160,004 ≈ 400.01.
The distance between B and F is approximately 400.01.

Step 5: Form the clusters.
We look for groups where the distances between members are small.
- Group 1 (Younger/Lower Spenders): Customers A, B, and F.
  - Distance A-B: 500.02
  - Distance A-F: 100.04
  - Distance B-F: 400.01
  These distances are relatively small compared to the spending scale. They form a tight cluster.

- Group 2 (Older/Higher Spenders): Customers C, D, and E.
  Let's check the distance between C and D:
  - Difference in age: 35 - 40 = -5. Squared: 25.
  - Difference in spending: 3000 - 4000 = -1000. Squared: 1,000,000.
  - Sum of squares: 1,000,025.
  - Square root ≈ 1000.01.
  
  Distance between D and E:
  - Difference in age: 40 - 45 = -5. Squared: 25.
  - Difference in spending: 4000 - 4500 = -500. Squared: 250,000.
  - Sum of squares: 250,025.
  - Square root ≈ 500.02.

  Distance between C and E:
  - Difference in age: 35 - 45 = -10. Squared: 100.
  - Difference in spending: 3000 - 4500 = -1500. Squared: 2,250,000.
  - Sum of squares: 2,250,100.
  - Square root ≈ 1500.03.

  The distances within Group 2 are larger (500 to 1500) than within Group 1 (100 to 500), but they are still distinct from Group 1. For example, the distance between A (Group 1) and C (Group 2) would be:
  - Age diff: 35-25=10 (sq 100)
  - Spend diff: 3000-2000=1000 (sq 1,000,000)
  - Total sq: 1,000,100
  - Distance ≈ 1000.05.
  
  Since the distance between groups is generally larger than the distance within Group 1, and the internal structure of Group 2 is consistent with higher values, we identify two clusters:
  Cluster 1: A, B, F
  Cluster 2: C, D, E

Step 6: Check for anomalies.
An anomaly would be a point that is very far from its own cluster center. 
- In Cluster 1 (A, B, F), the points are within 500 units of each other.
- In Cluster 2 (C, D, E), the points are within 1500 units of each other.
None of these points is extremely far from its neighbors. For instance, Customer C is about 1000 units from D and 1500 units from E. This is consistent with the spread of the group. There are no obvious anomalies in this small dataset.

Result:
1. Association: There is a strong positive association between age and spending (5 out of 6 customers follow the trend).
2. Clustering: We identified two natural groups: Younger/Lower Spenders (A, B, F) and Older/Higher Spenders (C, D, E).
3. Anomalies: None detected.

The mathematics used here is simple arithmetic: averaging, counting, subtracting, squaring, adding, and square roots. These same steps are performed by computers on millions of rows of data.


## From scratch, in Python

```python
# Data Mining from Scratch: discover patterns without libraries

# We will use plain Python lists. A list holds values in order, like a row of numbered boxes.
# Here is a list of ages:
ages = [25, 30, 35, 40, 45, 28]

# And a list of spendings:
spendings = [2000, 2500, 3000, 4000, 4500, 2100]

# 1. Association: count how many points agree in direction around the mean
# First, compute the mean (average) of a list.
def mean(numbers):
    total = 0
    for num in numbers:
        total = total + num
    return total / len(numbers)

mean_age = mean(ages)
mean_spending = mean(spendings)

# Count agreement: points where both age and spending are above their means, or both below.
agreement = 0
total_points = len(ages)
for i in range(total_points):
    if (ages[i] > mean_age and spendings[i] > mean_spending) or (ages[i] < mean_age and spendings[i] < mean_spending):
        agreement = agreement + 1

# The strength of association is the proportion of points that agree.
association_strength = agreement / total_points
print("Association strength (proportion of agreement):", association_strength)

# 2. Clustering: group points by distance. We'll do a simple approach: put each point in the cluster of its nearest 'seed'.
# We will just manually define two seeds based on the visual groups.
seed1 = [25, 2000]  # representative of young low-spenders
seed2 = [45, 4500]  # representative of older high-spenders

# Euclidean distance between two points, each a list of two numbers.
def distance(point1, point2):
    # squared difference in first coordinate
    diff1 = point1[0] - point2[0]
    # squared difference in second coordinate
    diff2 = point1[1] - point2[1]
    return (diff1**2 + diff2**2)**0.5

# Now assign each customer to the nearest seed.
clusters = [[], []]  # two empty lists for cluster 0 and cluster 1
for i in range(total_points):
    point = [ages[i], spendings[i]]
    dist_to_seed1 = distance(point, seed1)
    dist_to_seed2 = distance(point, seed2)
    if dist_to_seed1 < dist_to_seed2:
        clusters[0].append((ages[i], spendings[i]))
    else:
        clusters[1].append((ages[i], spendings[i]))

print("Cluster 0 (young low-spenders):", clusters[0])
print("Cluster 1 (older high-spenders):", clusters[1])

# 3. Anomaly detection: flag any point that is much farther from its cluster's centre than the rest.
# For simplicity, we compute the average distance from each point to its cluster centre, then check for outliers.

# Helper to compute centre (mean) of a list of points
def cluster_centre(points):
    if not points:
        return [0, 0]
    sum_x = 0
    sum_y = 0
    for p in points:
        sum_x = sum_x + p[0]
        sum_y = sum_y + p[1]
    n = len(points)
    return [sum_x / n, sum_y / n]

# Compute centres
centre0 = cluster_centre(clusters[0])
centre1 = cluster_centre(clusters[1])

# For each point, find its distance to its own centre
for cluster, centre in [(clusters[0], centre0), (clusters[1], centre1)]:
    distances = [distance(p, centre) for p in cluster]
    avg_dist = sum(distances) / len(distances)
    for p, d in zip(cluster, distances):
        if d > 1.5 * avg_dist:  # rule of thumb: more than 1.5 times the average is suspicious
            print("Anomaly found:", p, "distance", d)

# This is data mining in a nutshell: count, compare, and flag.
```


## The same thing, with a framework

```python
# With a library like pandas and scikit-learn, the same steps are minified.
# pandas handles the table of data, scikit-learn provides ready-made functions.

import pandas as pd
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

# Build a DataFrame (a table with named columns)
df = pd.DataFrame({'age': [25, 30, 35, 40, 45, 28],
                   'spending': [2000, 2500, 3000, 4000, 4500, 2100]})

# Standardise the features so that spending does not dominate distance (crucial!)
scaler = StandardScaler()
scaled = scaler.fit_transform(df)

# Perform K-Means clustering (more sophisticated than our manual seeds)
kmeans = KMeans(n_clusters=2, random_state=42, n_init=10)
clusters = kmeans.fit_predict(scaled)
df['cluster'] = clusters
print(df)

# Correlation as an association measure
correlation = df['age'].corr(df['spending'])
print("Correlation between age and spending:", correlation)

# Anomaly detection via distance to cluster centre
from sklearn.metrics import pairwise_distances
centres = kmeans.cluster_centers_
for i, row in enumerate(scaled):
    centre = centres[clusters[i]]
    dist = pairwise_distances([row], [centre])[0][0]
    if dist > 1.5:  # threshold chosen after inspecting values
        print(f"Anomaly: row {i} with distance {dist}")

# What the library does underneath: it runs the same arithmetic — means, distances, and comparisons —
# but on optimised C code that is much faster on large datasets.
```


## What the code does

We start with two lists: ages and spendings. A list holds values in order, and you access them by position (ages[0] is 25). The function mean() takes a list, loops through every number, adds them up, and divides by how many there are. The for loop is a way to do something to every item in a list. In the association step, we loop through the indices 0 to 5, and for each pair we ask: is age above its mean AND spending above its mean, or both below? If yes, we increment the agreement counter. The association strength is the fraction of points that agree — here it is 5/6 ≈ 0.83, a strong association. For clustering, we define two 'seed' points that we think represent two groups. The distance function computes the Euclidean distance, which uses the square root of the sum of squared differences — exactly the formula we did by hand. We then loop through each point, compute distance to both seeds, and assign to the nearest. The clusters list holds the assigned points as tuples (a pair of numbers). For anomaly detection, we compute the centre of each cluster by averaging the points, then compute each point's distance to that centre. If a distance is more than 1.5 times the average distance for that cluster, we flag it. In this small dataset, no point qualifies. The framework version shows how a data scientist would actually do it: pandas to hold the table, scikit-learn's KMeans to cluster, and built-in correlation for association. The key difference is that the library scales the features first, because spending values are hundreds of times larger than age values, and distance would be dominated by spending. That is a critical lesson you will see again in K-Nearest Neighbours.


## Common misconceptions

**Data mining is the same as machine learning** — If I know data mining, I know machine learning.

Data mining focuses on discovering patterns without a prior hypothesis, while machine learning usually starts with a specific target to predict. You can do data mining without building a predictive model, and you can train a model without ever exploring patterns in the data. In fact: Data mining is the exploratory part of the pipeline; machine learning is the predictive part. They overlap but are not the same.

**A pattern in a large dataset is always real** — If I find a strong association in my data, it must be true for everyone.

With millions of data points, chance alone will produce many apparent patterns. Without statistical testing or validation on a separate dataset, you cannot know if a pattern is real or just a coincidence. In fact: A pattern in the data is a candidate; you need to test it on a different sample to see if it holds.

**Clustering just groups obvious points** — Clustering is easy; the groups are obvious from a scatter plot.

In high dimensions (many features), you cannot see the data, and the notion of 'distance' becomes strange. Clustering algorithms choose the groups using a specific definition of distance, and different distances give different clusters. In fact: Clustering is a mathematical optimisation: it finds groups that minimise within-group distance, but the result depends on your distance choice and the number of groups.

**Anomaly means error** — An anomaly is always a mistake in the data.

An anomaly is just a point that is far from the rest. It could be a data entry error, but it could also be a genuine rare event — like a fraudulent transaction or a new disease symptom. In fact: An anomaly is a flag that something is unusual; you must investigate its cause before deciding what to do.

**Scale does not matter for distance** — I can compute distances directly on my raw numbers, no need to standardise.

If one feature ranges from 0 to 1000 and another from 0 to 1, the distance is almost entirely determined by the large-number feature, hiding the effect of the small one. In fact: Always scale features (for example, subtract the mean and divide by the standard deviation) before computing distances.


## In short

Data mining is the art of discovering patterns in large data without a prior hypothesis. The three classic tasks are association (what goes together), clustering (what forms a group), and anomaly detection (what is out of place). The mathematics is simple — means, counts, and distances — but the discipline lies in testing patterns so you do not mistake coincidence for signal. Scale your data before computing distances, and always validate findings on unseen data. This is the foundation for every model you will learn next.


## Where this is used

Data mining is everywhere in modern AI, though the name often changes. In large language models like GPT, the first step to building the model is mining the training corpus. The developers run association analyses to find which words tend to appear near each other — that is how the model learns that 'king' and 'queen' are related. They also run anomaly detection to remove toxic or duplicate content from the training data. In practice, a developer might use tools like TF-IDF or word co-occurrence matrices to quantify associations, and then feed those to a transformer. Underneath, it is the same counting and comparing we did here, just on billions of words. In small language models that run on a phone, the constraints are memory and speed. Data mining is used to prune the vocabulary: find the most common word patterns that cover most of the language, and drop the rest. This is association mining again — find which words are predictable from others, so the model does not need to store full sentences. Retrieval-augmented generation (RAG) is built on the same foundation. When you ask a chatbot a question, it needs to find relevant chunks of documents. That is a clustering and similarity search task: the system has already clustered all documents by embedding, and it finds the chunk whose cluster centre is nearest to your question's embedding. The code a developer writes uses a vector database like Pinecone or FAISS, which is an optimised distance computation engine — the Euclidean distance we wrote by hand, scaled to millions of points. Agentic systems that plan and call tools use data mining in a different way: they mine the logs of past actions to find patterns in what works. If an agent consistently fails when it calls a certain API, anomaly detection flags that failure mode, and association mining finds that a particular sequence of tool calls leads to success. The user's own interaction data is mined to cluster intents, so the agent can anticipate what the user wants next. In ordinary applied work, every churn prediction model starts with a data mining step to explore the data — find which customer attributes are associated with churn, cluster customers into segments, and flag suspicious data points. The developer writes pandas code to compute correlations and scatter plots, just like we did numerically. Data mining is the exploratory phase that decides which features go into the model. If it is done badly — for instance, running clustering without scaling features, or reporting associations without checking for coincidence — the model will be built on sand. A common failure: a churn model that finds a strong pattern in the training data that vanishes in production, because the data miner did not check whether the pattern was spurious. That is why the caution from this page — pattern is not proof — is the most important lesson. In vision applications like crop disease detection, data mining is used to find clusters of image features that correspond to healthy versus diseased leaves, and anomaly detection flags unusual leaf images that might be a new disease. The developer typically uses a clustering algorithm like K-Means on the output of a pre-trained CNN, and then inspects the clusters. The takeaway is that every AI system, no matter how complex, begins with the three tasks from this page: find what goes together, find the groups, and find what is odd.


---

Read this on the web: https://basavapurushottam.com/learn-ai/data-mining
