# Found Data

Data generated for another purpose and repurposed for analysis, as opposed to data collected by design.

**Rests on:** Unstructured Data

*Data Engineering and Analytics · Data Quality*


## Why learn this

You rely on numbers to make decisions every day, but most of those numbers were never intended for you. They were captured by sales teams recording purchases, or by servers tracking user clicks, or by sensors measuring temperature for equipment safety. When you use these numbers for your own goals, you are using 'found data'. Learning to distinguish between data designed for analysis and data found lying around is what allows you to judge whether your conclusions are real or just artifacts of how the records were kept. Without this distinction, every later decision about missing values, unusual entries, or system errors has no foundation.


## The intuition

Consider the difference between a diary you write to understand your own mind and an email you send to a friend. The diary is designed specifically to record your inner thoughts. The email was written to communicate a message, but if you read it years later, you might still learn something about what you were worrying about at the time. Found data is like that email: it was never created to be analyzed. It was created to run a business, to complete a sale, or to post on social media. Someone later says, 'These emails tell me about anxiety patterns,' or 'These sales records tell me about customer habits.' The data was found, not designed for that question. That origin leaves a mark on everything you do with it. The person who wrote the email had no idea you would read it for anxiety signals, so they did not write their true feelings clearly; they wrote what they wanted the recipient to see. Similarly, the salesperson who recorded a discount did not know you would later use that to predict demand; they recorded what the software required them to enter. The data carries the fingerprints of its original purpose, and you have to learn to see those fingerprints before you trust anything it tells you.


## An analogy

Imagine a city's surveillance cameras. They were installed to catch traffic violations, to monitor crowds, and to deter crime. Nobody installed them to study how people move through the city at different times of day. But a city planner comes along and says, 'I can use these camera feeds to understand pedestrian flow and design better sidewalks.' This is found data. The cameras now serve a second purpose they were never designed for. The crucial thing is that the way the cameras were placed biases what they see. They are dense in commercial areas and sparse in residential zones. A planner who forgets this will conclude that no one walks in residential areas, but that is an artifact of camera placement, not a fact about walking. The same happens in data. A social media platform records every click on the 'like' button but does not record the time someone hovers over a post without clicking. If you study engagement using only likes, you miss all the silent readers. The record was built for the platform's needs, not yours.


## Definition

Found data is information that was generated for an original operational purpose—such as running a business or recording a transaction—and is later repurposed for analysis, carrying the biases of its original collection process into every conclusion drawn from it.


## How it connects

This concept sits beside Data Cleaning, which deals with the mess in real datasets like missing values and duplicates that often results from data being found rather than designed. It also connects to Data Validation Constraints, which are the rules you encode to handle expectations that found data so often violates.


## The mathematics

Found data is not a calculation; it is a way of understanding where numbers come from. But there is one mathematical idea underneath it, from the branch called probability and statistics: sampling bias. That idea is about how the way you collect a set of numbers can make the set unrepresentative of the whole. The branch matters because statistics is the study of using a part to say something about the whole, and sampling bias is the specific failure mode where the part does not represent the whole.

The operation here is not a formula but a question: given a dataset, is it a fair sample of the population you care about? To answer that, you need to imagine the original process that produced the data. A process that selects some things and not others — a camera that points at intersections, a log that records clicks but not hovers — creates bias. The mathematics of bias is not a number you compute; it is a direction you reason about. You ask: what does this dataset over-represent, what does it under-represent, and why? The 'why' comes from understanding the original purpose.

There is no single formula for found data, because it is a property of the data's origin, not its content. The statistical concept it leads to is sampling bias. In formal statistics, we describe this error using the idea of an "estimate" and a "true value." An estimate is the number you calculate from your sample (like the 40% in the example below). The true value is the actual number in the whole population (which you usually cannot know). We write the average of many estimates as E[estimate]. If the sampling process is fair, the average of your estimates will equal the true value. If the sampling process is biased, the average of your estimates will not equal the true value. We write this as:

E[estimate] minus true_value equals zero (if fair)
or
equals some non-zero number (if biased).

But the useful 'formula' for you is a checklist: What was the original purpose? What was included? What was excluded? For what question is this data fair? For what questions is it biased?


## A worked example

```text
Suppose a food delivery app logs every order that is placed, including the time of day, the dish, and the customer's rating. You want to know how popular vegetarian food is. You take all orders from the last month and find that 40% of them were vegetarian. You conclude that 40% of the city's food choices are vegetarian.

Step 1: 40% of orders were vegetarian   # This is the number found in the app's records.
Step 2: But the app only delivers from restaurants that have signed up   # The app's list of restaurants is not the city's full list. Some restaurants, many of them small and possibly very vegetarian-friendly, are not on the app.
Step 3: Also, the app's users are younger and more urban than the city average   # The people who use the app are not a random sample of all eaters.
Step 4: The 40% is a property of the app's data, not the city   # If you had surveyed every restaurant or every person, you might get a very different number.

Result: The number 40% is not wrong as a fact about the recorded orders — it is exactly right. It is wrong as a fact about the city. The original purpose of the app's data (managing deliveries) shaped which data was collected (only certain restaurants, certain customers), and that shaping biases every statistic you compute from it.
```


## Common misconceptions

**Found data is bad data** — If data wasn't designed for my analysis, it's useless.

Found data is the only data most of us will ever have. It is abundant and cheap. The problem is not the data itself but the failure to account for its origin. In fact: Found data is usable; it just requires understanding and mitigating its biases. The bias is a feature to be recognised, not a reason to discard the data.

**The more data, the better** — If I have millions of rows, the data must be representative.

Found data can be enormous and still biased. A billion rows of clicks from one demographic is still just that demographic. In fact: Size does not cure bias. What matters is the alignment between the collection process and your question.

**The number is what it is** — The app said 40% vegetarian, so 40% of the city prefers vegetarian food.

The number is true only for the data that was collected, which came from a specific subset of restaurants and customers. In fact: A statistic is always a statement about the data you have, not about the world. You must infer the world only after accounting for how the data was produced.

**I can tell bias by looking at the data** — If I inspect the dataset carefully enough, I'll see what's wrong with it.

Bias is about what is not in the data. You cannot see the missing rows. The camera pointed at intersections does not show you the empty streets. In fact: Bias must be reasoned about from the collection process, not detected by looking at the data itself.

**Bias means unfairness** — When people say data has bias, they mean it's sexist or racist.

That is one important outcome of bias, but the word here means statistical bias: a systematic difference between what is measured and the truth. In fact: Statistical bias is the generic term. Unfairness is one possible consequence. The underlying issue is always the mismatch between collection and question.


## In short

Found data is abundant, cheap, and biased by its origin. Every dataset you will ever use was generated for someone else's purpose, and that purpose shapes what is included, what is missing, and what the numbers mean. Before drawing any conclusion, ask: why was this data created? What did the original collector care about? For what question is this data fair? For what questions is it blind? The answers are the difference between a model that works and one that confidently answers a question nobody asked.


## Where this is used

Found data is the default in real AI work — almost nothing you build will start from a survey you designed. In large language models, the training corpus is found data: the entire open web, scraped and repurposed. The original purpose of a Wikipedia article was to inform a human reader; the original purpose of a Reddit thread was conversation. Those purposes shape the model. The writing style, the factual focus, the biases of who contributed — all of that is baked into the model. When a developer says 'the model has a bias', they are looking at the residue of found data. In small language models running on a phone, the same training data is used but distilled, and the distillation process itself selects what to keep, which is another layer of found data. In retrieval-augmented generation, the system searches a database of documents — often internal company documents that were written for employees, not for AI retrieval — and the search indexes whatever was there. The original format, the jargon, the missing context all affect what the model retrieves. In agentic systems that plan and call tools, the agent acts on data from the user's environment: their calendar, their email, their files. That data was created for human scheduling and communication, not for an agent to parse, so the agent's reliability depends on how well it handles the quirks of that found data. In everyday applied machine learning — recommendation systems, fraud detection, forecasting — the training data is almost always found: past purchases, past transactions, past web sessions. A developer building a recommendation system writes Python to load that data, maybe with Pandas, and has to decide how to treat the fact that the data was logged by the company's servers without any thought to the recommendation question. What breaks when it is done badly: a model trained on found data can be silently wrong. The classic case is a model trained on customer support tickets that were filed only by angry users — it learns that all interactions are angry. The developer who does not ask 'why does this data exist?' will build a model that answers the wrong question with confidence.


---

Read this on the web: https://basavapurushottam.com/learn-ai/found-data
