# Entity Resolution

Determining when different records refer to the same real-world entity, across systems with no shared identifier.

**Rests on:** Data Mining

*Data Engineering and Analytics · Data Engineering*


## Why learn this

You will learn how to recognize that two separate rows of information actually belong to the same person, company, or product — even when those rows come from different software systems and share no common identification number. This skill is the key to fixing duplicate customer entries in a sales database, linking a bank’s internal records with external credit reports, and combining data after one business acquires another. Without this step, your reports will show inflated customer counts, missed opportunities for personalized service, and confused financial records. It matters now because clean, unified data is the only way to trust any analysis or automated process that relies on it.


## The intuition

Think of the last time you tried to find a friend on a social network or a professional directory. You typed a name and received several profiles as results. You had to look at the details to decide which one was actually them. Maybe one result had the correct name but a profile photo from a different city, suggesting it was someone else. Another result had the same job title and company but no profile photo, making it ambiguous. A third result matched the name and the city perfectly but showed a different last name because the person had recently changed their marital status. You combined these clues — name similarity, location match, professional history — to make a judgment call. Entity resolution is simply automating that specific judgment call for millions of records at once.

It is the process of deciding when two rows of data refer to the same real-world thing, like a person, a company, or a product, when neither row carries a unique identifier you can just match on directly. In an ideal world, every system would use one standard ID, like a national identity number for people or a tax identification number for businesses. In reality, the same company might be listed as 'Reliance Industries' in one database, 'RIL' in another, and 'Reliance Industries Ltd.' in a third. The same customer might be listed as 'A. Sharma' in one system and 'Anita Sharma' in another, with a different phone number because they moved house. Entity resolution is the discipline of bringing those records together without a shared key by comparing the fields you do have and scoring how likely it is that they are the same thing.


## An analogy

Imagine a busy railway station in a city like Mumbai, where a single person has bought tickets from three different counters, each run by a separate company. The first ticket says 'A. Sharma, Phone 98765 43210'. The second says 'Anita Sharma, Phone 98760 43210' — a typo in the last digit. The third says 'Anita S., Address: Malad West'. There is no ticket number shared across all three, so the system cannot just match on that identifier. A human clerk at a single desk, if they had access to all three, could look at the name, the phone number, and the address, and say, 'These are clearly the same person, let me merge them.' But if the station handled a million passengers a day, no team of clerks could do this by eye. Entity resolution is the automated clerk. It looks at pairs of records, compares each field — name, phone, address — and assigns a similarity score to each comparison. Then it combines those field-level scores into an overall score, and decides whether the records match, are definitely different, or are too close to call and need a human to check.

The analogy holds well because the clerk uses common sense, not an exact key. It breaks down in two crucial ways. First, a clerk can understand context: they know 'S.' is a common abbreviation for 'Sharma'. An automated system does not, unless you teach it. Second, a clerk can handle a handful of records; the station handles a million. The automated clerk must compare records without checking every possible pair, because a million records would mean five hundred billion comparisons, which would take too long. To solve this, the system uses 'blocking': a technique where records are grouped into smaller buckets based on loose criteria (for example, only comparing names that start with the same letter) so it never has to check pairs that could possibly be different.

The analogy stops working when you ask the clerk to do the impossible: check every pair. Automated systems must be smarter about which pairs to look at.


## Definition

Entity resolution is the process of determining when different data records refer to the same real-world entity, in the absence of a shared unique identifier, by comparing the fields in the records and deciding whether they match.


## How it connects

This topic sits inside your Data Engineering note, which you already have. It connects directly to defining strict rules for incoming data: when teams agree on exactly which fields must be populated and how they should be formatted, entity resolution becomes reliable because there is clear information to compare. It also connects to viewing all your data as one unified map: when you know that 'customer 42' in one system is the same person as 'customer 73' in another, you can merge queries and see the complete history of that customer rather than fragmented glimpses.


## The mathematics

The mathematics here is not calculus or linear algebra; it is about comparing numbers and computing an overall similarity score. You will need arithmetic — adding, multiplying, and taking ratios — and a simple idea from algebra called a weighted sum. You might think you do not know what a weighted sum is, but you do: it is like computing your final grade when the exam is worth 60% and the homework is worth 40%. The weights are the percentages, and the weighted sum is the total. In entity resolution, each field comparison (name, phone, address) gets a similarity score, and each comparison gets a weight that says how important it is. The weighted sum of those scores gives you an overall match score. For instance, if the name matches perfectly, that might give a similarity of 1.0, and the phone almost matches, giving 0.9. If the name has weight 0.7 and the phone has weight 0.3, the overall score is 0.7 × 1.0 + 0.3 × 0.9 = 0.97. That is the mathematics: a weighted sum, exactly like a final grade. You will also meet two ways to compare text fields: exact match, which is either 0 or 1, and a ratio-based measure like the Jaccard similarity, which counts how many letters or words are shared between two strings relative to how many they have in total. That is arithmetic too: dividing one count by another. There is no derivative or matrix here; this concept is about combining simple numbers in a sensible way.

Match Score = Σᵢ (Weightᵢ × Similarityᵢ)

Where:
- Σᵢ means 'sum over all fields i' (e.g., name, phone, city)
- Weightᵢ is a number between 0 and 1, chosen to reflect how important field i is, with all weights summing to 1
- Similarityᵢ is the similarity score for field i: 1 for exact match, 0 for exact non-match, or a fractional value between 0 and 1 for a partial match (like a phone number with one typo)

The name of this branch is a weighted sum, and the core operation is the dot product, which you will meet properly in the next topic on embeddings. Here, you just multiply matching pairs and add the results.


## A worked example

```
We have two records for what might be the same person. Record A: Name 'Anita Sharma', Phone '98765 43210', City 'Mumbai'. Record B: Name 'Anita Sharma', Phone '98765 43210', City 'Mumbai'. We compare each field and assign a similarity score of 1 if exact match, 0 otherwise. The weights we have chosen (as a human, or from training data) are: Name weight 0.5, Phone weight 0.3, City weight 0.2. These weights sum to 1.0.

Step 1: Name: 'Anita Sharma' vs 'Anita Sharma' → exact match → similarity = 1.0   # The names are exactly the same, so the similarity score is 1.
Step 2: Phone: '98765 43210' vs '98765 43210' → exact match → similarity = 1.0   # The phone numbers are also exactly the same.
Step 3: City: 'Mumbai' vs 'Mumbai' → exact match → similarity = 1.0   # The cities match exactly as well.
Step 4: Weighted sum = (0.5 × 1.0) + (0.3 × 1.0) + (0.2 × 1.0) = 0.5 + 0.3 + 0.2 = 1.0   # We multiply each similarity by its weight and add them up.

Result: The overall match score is 1.0, meaning a perfect match. If our threshold for 'match' is 0.8 or above, we declare these records the same person. If the threshold were 0.95, we would not declare a match, which shows how the threshold choice changes the outcome.
```


## From scratch, in Python

```python
# We compare two records field by field.
# A record is a dictionary: keys are field names, values are the data.
# A dictionary is a Python structure that stores values by a key, like a name tag.

record_a = {"name": "Anita Sharma", "phone": "98765 43210", "city": "Mumbai"}
record_b = {"name": "Anita Sharma", "phone": "98765 43210", "city": "Mumbai"}

# A dict to hold the weights for each field.
weights = {"name": 0.5, "phone": 0.3, "city": 0.2}

# We need a function to compare two strings and return a similarity between 0 and 1.
# A function is a reusable block of code that takes input, does something, and gives output.

def similarity(a, b):
    if a == b:
        return 1.0
    else:
        return 0.0

# We start the total score at 0.
total_score = 0.0

# We loop over each field in the weights dictionary.
# A loop repeats a block of code for each item in a collection.
for field in weights:
    # Get the similarity for this field.
    sim = similarity(record_a[field], record_b[field])
    # Multiply by the weight and add to the total.
    total_score = total_score + (weights[field] * sim)

print("Match score:", total_score)

# Output: Match score: 1.0

# Now let's try a pair with slight differences.
record_c = {"name": "Anita Sharma", "phone": "98760 43210", "city": "Mumbai"}
# We could update the similarity function to be smarter about one-digit typos,
# but for now it will return 0 for the phone.

# Reuse the same loop, but this time compare A and C.
total_score = 0.0
for field in weights:
    sim = similarity(record_a[field], record_c[field])
    total_score = total_score + (weights[field] * sim)

print("Match score for A vs C:", total_score)

# Output: Match score for A vs C: 0.7
```


## The same thing, with a framework

A common library for fuzzy string matching is `rapidfuzz` or `fuzzywuzzy`. Here is how you would use `rapidfuzz` to get a partial similarity for names and phones:

```python
# pip install rapidfuzz
from rapidfuzz import fuzz

# fuzz.ratio gives a score between 0 and 100 for how similar two strings are.
ratio_name = fuzz.ratio("Anita Sharma", "Anita Sharma") / 100.0  # 1.0
ratio_phone = fuzz.ratio("98765 43210", "98760 43210") / 100.0  # ~0.9

# You can then use these scores in the same weighted sum.
```

Under the hood, `fuzz.ratio` compares the two strings character by character, counts how many characters are the same in order, and divides by the total characters. It is the same kind of counting and dividing arithmetic you did by hand, just done fast in C.


## What the code does

The code defines two records as Python dictionaries, which pair a field name with a value — like a row in a spreadsheet with column headers. The `weights` dictionary holds the importance of each field. The `similarity` function returns 1.0 when the strings are identical, 0.0 otherwise. A `for` loop iterates over each field name in the weights dictionary, looks up the value in both records, computes the similarity, multiplies by the weight, and adds it to the running total. The output is the weighted sum. This is the exact arithmetic from the mathematics section. For the second pair, the phone differs, so the phone contribution is 0, and the final score is 0.5 × 1.0 + 0.3 × 0.0 + 0.2 × 1.0 = 0.7. This plain code shows how the loop and the weighted sum work. It does not yet handle partial matches like Jaccard similarity — that is the next step, and a library can help.


## Common misconceptions

**Exact match on name is enough** — If I just match on the name, I will catch all duplicates.

Names have typos, abbreviations, honorifics, spelling variations, and even the same name can belong to different people. A name match alone catches the obvious cases and misses the rest, and it also creates false matches when two different people share a name. In fact: Name is one weak signal among many. A robust resolution uses several fields — name, phone, address, date of birth — and weights them, because no single field is reliable on its own.

**Entity resolution is a one-time cleanup task** — I clean my database once, deduplicate it, and it stays clean forever.

New records arrive, people change addresses, companies change names, systems get migrated. The matching thresholds that worked last year may no longer fit the data distribution. The problem is continuous, not a one-off job. In fact: Entity resolution must be a regular step in the data pipeline, run on every new batch, with thresholds reviewed and the model retrained when the data shifts.

**More fields always make the match better** — Adding more fields to compare will always improve the score.

Some fields are noisy, some are missing, and some are worthless for discrimination. For example, comparing 'city' adds nothing when every record is from Mumbai. Weighing a noisy field too highly can distort the overall score and cause false matches or false non-matches. In fact: Choose fields that are stable, informative, and independently recorded. Discard or down-weight fields that are either empty or almost always the same. The weights matter as much as the fields themselves.

**Bias means the model is unfair** — The 'bias' term in the weighted sum formula is bad, like racial bias.

Here the word 'bias' is unrelated to fairness. In entity resolution the term is often not used, but when it appears it refers to a constant offset added to a score, shifting the threshold. It is a technical term, not a moral judgment. In fact: In this context, bias is a constant added to a score to adjust for base rates; it has no ethical connotation.


## In short

Entity resolution is the process of deciding when two records refer to the same real-world entity without a shared ID. It works by comparing available fields, turning each comparison into a similarity score, and combining those scores into a weighted sum that is judged against a threshold. It is genuinely hard because data is messy and the problem is continuous, and it is tractable only with a blocking strategy that reduces the number of candidate pairs. Precision and recall trade off directly: a higher threshold reduces false matches but increases missed ones, and the right balance depends on the cost of each error in the application.


## Where this is used

Entity resolution is not a single trick but a whole set of techniques, and it shows up in every corner of building AI systems. In an LLM-based system, entity resolution is often used in the data-cleaning stage before training or fine-tuning: if you scrape data from many websites, the same company might appear as 'Acme Corp', 'Acme Inc', and 'Acme'. If you do not resolve these into a single entity, the model may learn that 'Acme Inc' and 'Acme Corp' are different companies, or worse, that 'Acme' is a third one. A developer using an LLM pipeline might write a script that extracts all company names, normalises them, and applies a fuzzy matching library to merge them before tokenising. In retrieval-augmented generation (RAG), entity resolution matters at query time: a user asks 'what is Reliance's revenue?', and the retrieval system must match that to the document that talks about 'Reliance Industries Limited'. If the embedding similarity is used alone, it may not distinguish between the telecom and petrochemical entities. A developer might build an entity index that maps all names to a canonical ID, and use that ID as a filter before the vector search. In agentic systems that plan and call tools, an agent often needs to combine information from a CRM and a support ticket system. The agent must decide that customer 'A. Sharma' in the CRM is the same as 'Anita Sharma' in the tickets. An agent that resolves this correctly can write a coherent reply; one that does not may quote the wrong history. The developer writes a tool or a model step that takes two records and returns a match confidence, and the agent uses that confidence to decide whether to merge. In small models running on devices, like a phone-based contact deduplicator, entity resolution has to run with minimal memory and processing. The developer uses simple rules: normalise names to lowercase, strip punctuation, and use a lightweight string comparison. A more complex model would be too slow, so they choose blocking and a simple exact-match on a few fields. In everyday applied machine learning, entity resolution is used in recommendation systems to deduplicate users before collaborative filtering, in fraud detection to link bank accounts that may belong to the same person, in forecasting to merge sales records from multiple store systems, in speech recognition to align a caller's name with the account database, and in vision to match the same object across camera frames. In all of these, the developer writes code that builds pairs of records, compares field by field, computes a similarity, and applies a threshold. The failure mode when done badly is either duplicate records that cause overcounting, or missed matches that cause undercounting — both silently corrupt metrics. The threshold is a business decision, and choosing it without reference to the cost of each error is how systems quietly betray their users. The mathematics is always the same weighted sum; the art is in the field-specific similarity functions and the blocking strategy that makes the problem tractable.


---

Read this on the web: https://basavapurushottam.com/learn-ai/entity-resolution
