# Data Transformation

Reshaping data into the form analysis requires — type conversion, normalisation, encoding, aggregation and pivoting.

**Rests on:** Data Cleaning

**Leads to:** DNA Sequencing Analysis, Unicode and Encoding Pitfalls

*Data Engineering and Analytics · Data Quality*


## Why learn this

Data transformation is the bridge between messy reality and analysis. Every serious AI project spends most of its time here, not on the model itself. Mastering transformation means you can: feed raw data into any model without errors, make different features comparable, turn text into numbers for machine learning, and reshape tables to answer different questions. It is the prerequisite for exploring data, building models, and deploying them reliably. Without it, the most elegant algorithm is useless. With it, you can adapt almost any dataset to almost any task.


## The intuition

Imagine you need to bake a cake, but the recipe is in metric and your measuring cups are imperial, the eggs are in a carton but the recipe calls for beaten eggs by weight, and the oven temperature is in Celsius while yours reads Fahrenheit. You wouldn't throw up your hands — you would convert units, beat the eggs, and preheat accordingly. That's data transformation: taking the ingredients as they come and converting them into the exact form the next step requires. In AI, data arrives in all sorts of shapes and types — numbers in text form, categories as words, values on wildly different scales, or spread across separate tables. Transformations fix these: convert strings to numbers, encode categories, scale values, and merge or reshape tables. The goal is always the same: make the data speak the language the analysis or model understands. If you skip this step, your model either fails outright or gives silently wrong answers. The same thinking applies to every transformation: "What does the next step expect, and how do I get there from what I have?"


## An analogy

Think of data transformation as a well-organized kitchen before a busy dinner service. The chef needs ingredients prepped, measured, and arranged in the order of use. You have a sack of potatoes, some chicken, a bag of onions, and a jug of stock. The prep cook washes, peels, chops — and crucially, writes down what they did: "Potatoes, 5 kg, peeled, cut into 2 cm cubes, at 4 pm." This note matters because the sous chef later needs to know exactly how much potato went into the stew to adjust seasoning. The analogy breaks in one important way: a kitchen is physical and forgiving — a slightly larger cube still cooks. Data transformation is digital and unforgiving: a wrong conversion can silently flip conclusions, and once data is transformed, you cannot always tell what the original looked like. That's why recording steps is essential, not optional. The kitchen analogy also underplays the scale: you might transform billions of rows, not a few dozen, and the note becomes the only way to ensure reproducibility — the same steps applied tomorrow should give the same result.


## Definition

Data transformation is the process of reshaping data from its raw, as-collected form into a structured, consistent form that is fit for analysis or modelling, through operations like type conversion, normalisation, encoding, aggregation, and pivoting.


## How it connects

This concept builds on nothing (you are starting here). It connects directly to your library notes on **Data Cleaning** (often done together — cleaning fixes errors, transformation changes shape), **Data Dictionaries** (transformations should be documented there), and **Handling Missing Data** (how you treat missing values can change what transformations are appropriate). It is a core part of **Data Quality** — a dataset that isn't transformed properly is not fit for its purpose. As you progress, transformation is a prerequisite for **Exploratory Data Analysis** and **Feature Engineering**. Your library notes emphasise that preprocessing is the majority of the work, and this page is where that work happens.


## The mathematics

Data transformation is the process of changing raw numbers or categories into a form that a computer can understand and use. Think of it like preparing ingredients before cooking: you wash, peel, chop, and measure them so they are ready for the recipe. In data work, we do four main things.

First, type conversion. Computers store text and numbers differently. A word like "forty-two" is just a sequence of letters. The number 42 is a value. Type conversion changes the text into the number. We read the characters, recognize they represent a quantity, and store that quantity as a number. This allows us to do math with them.

Second, normalization. Different features have different scales. One feature might be age (0 to 100). Another might be income (0 to 1,000,000). If we put these directly into a model, the large numbers will dominate the small ones. Normalization squeezes all values into a standard range, usually from 0 to 1. We do this by finding the smallest value in the set and the largest value. Then, for every number, we subtract the smallest value from it. This shifts the lowest number to 0. Next, we divide that result by the difference between the largest and smallest values (the range). This stretches or shrinks the numbers so the new highest value becomes exactly 1, and everything else falls in between.

Third, encoding. Computers cannot process words like "male" or "female" directly in most mathematical formulas. We assign a unique number to each category. For example, we might decide that "male" becomes 0 and "female" becomes 1. This is a simple lookup: find the word, return the assigned number.

Fourth, aggregation. Sometimes we need a single summary number for a group of values. The most common summary is the average (mean). To find the average, we add all the numbers in the group together. Then we count how many numbers there are. Finally, we divide the total sum by the count. This gives us the central value of the group.

Pivoting is another step, but it is about rearranging the table structure rather than changing the values themselves. It groups rows together and spreads them across columns. We will focus on the value changes above.

The formula for normalization is:

normalized_value = (original_value - minimum_value) divided by (maximum_value - minimum_value)

This formula works for every single number in your list.


## A worked example

We have three customers. Their ages are stored as text: '25', '34', and '41'. Their incomes are numbers in rupees: 50,000, 62,000, and 48,000.

Step 1: Type conversion of ages.
The computer sees the text string '25'. It converts this to the number 25. It sees '34' and converts it to 34. It sees '41' and converts it to 41. Now we have three numbers: 25, 34, 41.

Step 2: Identify the range for income normalization.
The incomes are 50,000, 62,000, and 48,000.
Find the minimum income: The smallest number is 48,000. So, minimum_value = 48,000.
Find the maximum income: The largest number is 62,000. So, maximum_value = 62,000.
Calculate the range: Subtract the minimum from the maximum. Range = 62,000 - 48,000 = 14,000.

Step 3: Normalize the first income (50,000).
Subtract the minimum from the original value: 50,000 - 48,000 = 2,000.
Divide by the range: 2,000 divided by 14,000.
Calculation: 2,000 / 14,000 = 0.142857...
Rounded to three decimal places, this is 0.143.

Step 4: Normalize the second income (62,000).
Subtract the minimum from the original value: 62,000 - 48,000 = 14,000.
Divide by the range: 14,000 divided by 14,000.
Calculation: 14,000 / 14,000 = 1.0.
This makes sense because 62,000 was the maximum value, so it maps to 1.

Step 5: Normalize the third income (48,000).
Subtract the minimum from the original value: 48,000 - 48,000 = 0.
Divide by the range: 0 divided by 14,000.
Calculation: 0 / 14,000 = 0.0.
This makes sense because 48,000 was the minimum value, so it maps to 0.

Result: The normalized incomes are [0.143, 1.0, 0.0]. All values are now between 0 and 1, allowing them to be compared fairly with other features.


## From scratch, in Python

```python
# Plain Python: type conversion, normalisation, encoding, aggregation, pivoting

# 1. Type conversion: from string to integer
ages_text = ['25', '34', '41']
ages = []
for age in ages_text:
    ages.append(int(age))  # int() converts string to number
print(ages)  # [25, 34, 41]

# 2. Normalisation: scale incomes to 0-1
incomes = [50000, 62000, 48000]
min_income = min(incomes)
max_income = max(incomes)
normalised = []
for income in incomes:
    norm = (income - min_income) / (max_income - min_income)
    normalised.append(norm)
print(normalised)  # [0.142857..., 1.0, 0.0]

# 3. Encoding: map categories to numbers
categories = ['male', 'female', 'male', 'female']
encoded = []
for cat in categories:
    if cat == 'male':
        encoded.append(1)
    else:
        encoded.append(0)
print(encoded)  # [1, 0, 1, 0]

# 4. Aggregation: average of ages
sum_ages = 0
for age in ages:
    sum_ages = sum_ages + age
mean_age = sum_ages / len(ages)
print(mean_age)  # 33.333...

# 5. Pivoting: from long to wide format
# Assume data: [customer, product, amount]
data = [
    ('A', 'apple', 5),
    ('A', 'banana', 3),
    ('B', 'apple', 2),
    ('B', 'banana', 4),
]
# Want wide: each customer is a row, each product is a column
customers = ['A', 'B']
products = ['apple', 'banana']
wide = {}
for customer in customers:
    row = {}
    for product in products:
        # find amount for this customer and product
        for entry in data:
            if entry[0] == customer and entry[1] == product:
                row[product] = entry[2]
                break
    wide[customer] = row
print(wide)
# {'A': {'apple': 5, 'banana': 3}, 'B': {'apple': 2, 'banana': 4}}
```


## The same thing, with a framework

```python
# Same operations in pandas (the standard library for data transformation)
import pandas as pd

# Type conversion
ages = pd.Series(['25', '34', '41']).astype(int)

# Normalisation
incomes = pd.Series([50000, 62000, 48000])
normalised = (incomes - incomes.min()) / (incomes.max() - incomes.min())

# Encoding (label encoding)
categories = pd.Series(['male', 'female', 'male', 'female'])
encoded = categories.map({'male': 1, 'female': 0})

# Aggregation
mean_age = ages.mean()

# Pivoting
df = pd.DataFrame({'customer': ['A','A','B','B'],
                   'product': ['apple','banana','apple','banana'],
                   'amount': [5,3,2,4]})
wide = df.pivot(index='customer', columns='product', values='amount')
```


## What the code does

The from-scratch code starts with a simple **list** — a collection of items in order. We loop through each item using a `for` loop, converting text to integers with `int()`. For normalisation, we first find the minimum and maximum using `min()` and `max()`, then compute each normalised value. Encoding uses an `if-else` statement to assign a numeric code. Aggregation sums with a loop, then divides by the length. Pivoting is the most complex: we iterate over customers and products, searching the list of tuples for each match. The pandas version does all this in one or two lines — `astype` converts, `.map` encodes, `.mean()` aggregates, and `.pivot` reshapes. Pandas is doing exactly the same underlying operations as your hand-written loops.


## Common misconceptions

**Transformation is a boring, trivial step** — The real work is the model; transforming data is just a chore I can rush through.

Most machine learning failures trace back to bad data, not bad models. A single un-encoded category or wrong normalisation can silently destroy accuracy. Your library notes say preprocessing is the majority of the work. In fact: Transformation is a core part of the work — it decides whether the model sees what it needs. Done carefully, it can even improve accuracy beyond algorithm choice.

**Normalisation is the same as standardisation** — Scaling to 0-1 is the same thing as making the mean 0 and standard deviation 1.

They are different methods. Min-max scaling gives a fixed 0-1 range; standardisation (z-score) has mean 0 and variance 1, but can go outside 0-1. Choosing the wrong one changes the model's behaviour. In fact: Min-max scaling is (x - min)/(max - min); standardisation is (x - mean)/std. Use min-max when you know the bounds; use standardisation when data might have outliers.

**Transformations can be done after the model is trained** — I can transform the test data separately just before prediction.

Transformations must be fit on the training data only, and then applied to the test data. If you fit on test data, you leak information. Your library also warns that transformations must be identical across train and serve. In fact: Learn parameters (like min, max, mean, std) from training data only, and apply the same function to test and serving data. Reproducibility is key.

**Type conversion is harmless — I can do it anywhere** — I can convert a string to a number at any point; it doesn't matter when.

If you convert too late, the model may have already seen text features and split them incorrectly. If you convert too early, you may miss catching non-numeric values that should be errors. In fact: Type conversion should happen early in the pipeline, right after ingestion, to ensure all downstream steps work on the correct types.

**Pivoting and aggregating are interchangeable** — Whether I reshape the table or just summarise it, I get the same answer.

Pivoting changes the shape without losing detail; aggregating collapses rows and loses detail. They answer different questions. In fact: Pivot to see each entity as a row with features as columns; aggregate to get summaries like averages per group. Use both as needed.

**Encoding always means assigning 0, 1, 2...** — I can just map categories to sequential integers — that's the standard way.

For nominal categories, sequential integers imply an order that doesn't exist. A model might interpret 2 as twice 1. One-hot encoding or similar is safer for nominal data. In fact: For nominal categories, use one-hot encoding (each category becomes a binary column). Ordinal categories can use integer codes respecting their order.


## In short

Data transformation is the essential reshaping of raw data into a form analysis needs. Key operations: type conversion, normalisation, encoding, aggregation, pivoting. The order matters and must be reproducible. Transformations must be identical across train and serve, and recording them is what makes results explicable. Most AI project time goes here, not to modelling.


## Where this is used

Data transformation is everywhere in real AI. In **large language models**, text must be converted into tokens (numbers) — that's a transformation of the same kind as encoding. Words are mapped to integer IDs, then to vectors — again a transformation. Without it, the model couldn't process text. In **small models on devices**, like keyword spotters on phones, audio is transformed into spectrograms (numbers) and then normalised — the same min-max scaling you did by hand. In **retrieval-augmented generation**, documents are chunked and embedded — embeddings transform text into vectors so they can be compared. The transformation is the embedding function. In **agentic systems**, tool outputs often come as JSON text — the agent must transform them into Python objects (type conversion) to reason about them. In **everyday machine learning**, almost every dataset needs some combination of encoding, normalisation, and reshaping before a model can touch it. A developer writes `pd.get_dummies()` for one-hot encoding, `sklearn.preprocessing.MinMaxScaler` for scaling, and `df.pivot` for reshaping. They rarely write these from scratch, but they must understand what's happening underneath — otherwise, a wrong transform silently corrupts results. When done badly, transformations cause bugs: mixing units, forgetting to encode a category, or normalising with incorrect min/max. These are common causes of model failure in production.


---

Read this on the web: https://basavapurushottam.com/learn-ai/data-transformation
