# Parquet and Flat-File Hygiene

Practices for durable file-based data — columnar formats, schema evolution, partitioning and predictable naming.

**Rests on:** Data Contracts

*Data Engineering and Analytics · Data Engineering*


## Why learn this

This is the foundation of every data pipeline. Before you can train a model, run a query, or build a data contract, your data must live in a file that is fast to read, safe to change, and cheap to filter. Master this and you unlock: **Data Contracts** (agreeing on schema and quality across teams), **Data Pipelines** (moving and transforming data reliably), **MLOps** (deploying and monitoring models that depend on stable inputs), and **Big Data Processing** (working with terabytes without grinding to a halt).


## The intuition

Think of your data as a stack of index cards. A **flat file** (like CSV) stores each card as one long line of text, with commas separating the fields. Every card has the same fields in the same order. If you want all the cards where 'age' is over 30, you must read every single line — there is no shortcut. A **columnar format** (like Parquet) stores the data differently: instead of one card per line, it groups all the 'age' values together, then all the 'name' values, and so on. It is like having a separate deck of cards for each attribute. To find everyone over 30, you only flip through the 'age' deck, not the whole box. This matters enormously when a file has millions of rows. Columnar formats also compress better — repeated values pack tightly — and they carry a **schema** (a description of each column's type) right inside the file, so when you read it back, the computer knows that 'age' is a number and 'name' is text. A CSV file has no such description; the reader has to guess, and often guesses wrong, turning '0123' into the number 123. Finally, **partitioning** means splitting the data into separate files or folders by a key, like date — so a query that only wants January reads just the January folder, not the whole year. This is the difference between a data pipeline that takes seconds and one that crawls for hours.


## An analogy

Imagine a library. A flat file is a single, enormous book with every fact crammed in order — row by row. To find 'books published after 2010', you must read every page. A columnar format is like splitting that book into separate volumes: one volume for publication year, one for title, one for author. Now you only pick up the 'publication year' volume and scan it — much faster. The library also has a **catalogue** (the schema) that tells you each volume's format: dates are ISO, numbers are integers, text is UTF-8. If a new edition adds a field (like 'awards'), the catalogue is updated, and volumes that predate the change have blanks — that is **schema evolution**, and it is a negotiation, not a crash. **Partitioning** is like a library with a separate room for each year — you walk into the 2010 room and ignore the rest. **Naming** is the shelf labels: if every file is called 'data1.parquet', you never know what is inside; if it is 'sales_2024_01.parquet', you do. But the analogy breaks: a library can skip a page when reading, but a columnar file, if you read one column, you must read the whole column — there is no random access to a single row. Also, columnar files are more complex to write; you need to buffer many rows before writing, so they are not ideal for streaming one record at a time. And in a library, the catalogue is separate from the books; in Parquet, the schema is embedded — the book carries its own catalogue.


## Definition

Parquet and flat-file hygiene is the discipline of storing tabular data in a way that is fast to read, unambiguous about its types, and easy to update — using columnar formats with embedded schemas, partitioning on high-value keys, and predictable naming that encodes what is inside.


## How it connects

You have not studied anything yet — this is your first step. But this concept directly supports what you will learn next: **Data Contracts** (the schema embedded in Parquet becomes the contract you agree on with other teams), **Data Pipelines** (your pipeline's output must be a clean, partitioned Parquet file), and **MLOps** (models depend on stable, typed features; a CSV that silently changes types can break a retraining job). In the other direction, this relies on nothing but basic file know-how — you are starting fresh, and that is fine.


## The mathematics

The mathematics here is simple arithmetic — counting and comparing sizes. When we say a columnar format is 'faster', we mean it reads fewer bytes. The arithmetic is just: **number of bytes to read = (number of values you need) × (size of each value)**. In a flat file, you need every row to find one column, so it is **total rows × total columns × average size**. In a columnar file, you only read the column you need, so it is **total rows × 1 column × size**. The ratio is the speedup. There is also arithmetic in **compression**: if a column has many repeated values (like a 'country' column full of 'India'), the file uses a trick called run-length encoding — it stores 'India appears 5000 times' instead of 'India, India, India...' 5000 times. Counting how many times a value repeats is literally counting — no algebra needed. **Schema evolution** uses no mathematics beyond understanding that adding a column means some rows have a blank — that is just a missing value, zero in arithmetic. Partitioning uses **division**: if you partition by year, a query filtered to 2024 reads 1/10 of the files if there are 10 years — but only if the data is evenly spread; that is a ratio, a simple fraction.

Bytes read = (rows_in_scope) × (columns_in_scope) × (avg_bytes_per_value)  
where:
- rows_in_scope = total rows if you must read sequentially, or rows in selected partitions if you filter by partition key
- columns_in_scope = total columns for flat, 1 for a single column read in columnar
- avg_bytes_per_value = raw or compressed size per value


## A worked example

```
You have a dataset with 1,000,000 (one million) rows and 50 columns, each value takes 10 bytes. You want to find the average 'age' column.

Step 1: Flat file: 1,000,000 rows × 50 columns × 10 bytes = 500,000,000 bytes (500 MB)   # You must read every row and every column to get to the 'age' column.
Step 2: Columnar file: 1,000,000 rows × 1 column × 10 bytes = 10,000,000 bytes (10 MB)   # You only read the 'age' value for each row — the columnar format stores it together.
Step 3: Speedup = 500 MB / 10 MB = 50 times   # That is why anyone working with large data uses columnar formats.
Step 4: Compression: 'country' column has 1,000,000 values, 80% are 'India', each value 5 bytes. Stored as repeated: 800,000 × (5 + 1 byte marker) ≈ 4.8 MB; without compression: 5,000,000 bytes (5 MB) — a modest saving, but if values are highly repetitive, it can be 10x.   # Run-length encoding stores 'value, count' pairs instead of repeating the value.
Step 5: Partitioning by year: 10 years, each with 100,000 rows. Query 'where year = 2024' reads 1/10 of the data = 10 MB instead of 100 MB.   # You skip the other partitions entirely; the cost is proportional to the fraction of data you need.

Result: Reading one column from a flat file costs 500 MB; from Parquet, 10 MB — a 50× speedup. Compression and partitioning multiply this further.
```


## From scratch, in Python

```python
# Python program to simulate row vs column storage and measure read cost.
# No frameworks — just lists and a loop.

# A list holds items in order. Here's a list of three names:
names = ["Asha", "Rahul", "Priya"]
# Each string is an item. You can access by position: names[0] is "Asha".

# A dataset as rows — each row is a list of values.
rows = [
    ["Asha", 25, "India"],
    ["Rahul", 30, "India"],
    ["Priya", 35, "Nepal"],
]

# Simulate reading the whole flat file: loop over every row, every column.
# "row" goes through each inner list. "value" goes through each item.
flat_bytes_read = 0
for row in rows:
    for value in row:
        # len(str(value)) gives how many bytes roughly for a string representation.
        flat_bytes_read += len(str(value)) + 1  # +1 for comma or newline

# Simulate reading just the 'age' column (index 1) — you still read all rows in flat format.
column_bytes_row = 0
for row in rows:
    column_bytes_row += len(str(row[1])) + 1

print("Flat file read (all columns):", flat_bytes_read)
print("Flat file read (only age):", column_bytes_row)

# Now simulate columnar storage: three separate lists, one per column.
ages = [25, 30, 35]
# To read the age column, you read only this list:
column_bytes_columnar = 0
for age in ages:
    column_bytes_columnar += len(str(age)) + 1

print("Columnar read (only age):", column_bytes_columnar)
print("Speedup (if only reading age):", flat_bytes_read / column_bytes_columnar)

```


## What the code does

This code demonstrates the core idea without any real file formats. A list is a collection of items in order — `names = ["Asha", "Rahul", "Priya"]` is three strings. You access elements by position: `names[0]` is the first item, "Asha". We make `rows` as a list of lists — each inner list is a row. The outer list holds all rows. A for loop goes through each item in a list. In `for row in rows`, the variable `row` takes each inner list in turn. In the inner loop, `for value in row`, it takes each value. We sum up lengths of string representations plus a comma, approximating bytes. For the columnar version, we create separate lists per column — `ages` holds just the age values. Reading `ages` is like reading a whole column. The speedup is the ratio of flat bytes to columnar bytes. Note: this is a simulation; real Parquet does binary, but the concept is the same — data for a column is stored together.


## Common misconceptions

**CSV is faster because it is simpler** — CSV is a plain text file, so it must be quicker to read and write.

CSV costs more I/O because it reads all columns when you only need one, and parsing text into numbers is CPU-heavy. Parquet is binary but skips whole column chunks; for any realistic dataset, the total time is far lower. In fact: Parquet's columnar layout and compression mean it is almost always faster to read — the 'simplicity' of CSV is on the surface, not in the reading.

**Partitioning makes everything faster** — If I partition by every possible column, all my queries will be quick.

Each partition adds overhead (directory entries, file headers). Partitioning on a column with many unique values (like customer ID) creates millions of tiny files — slower than one big file. In fact: Partition on low-cardinality, frequently-filtered keys like date or country. Too many small files is a known anti-pattern.

**Schema evolution is automatic** — If I add a new column to my Parquet file, everything will just work.

Adding a column is fine only if readers ignore it or handle missing values. Changing a type (e.g., from int to string) is a breaking change that requires migration or backfill. In fact: Schema evolution follows rules: additive changes are safe; destructive changes are a contract negotiation, not a silent update.

**Parquet is only for 'big data'** — My dataset is small, so I can stick with CSV and never worry.

Even with a thousand rows, type inference in CSV can misinterpret '001' as the number 1, corrupting an ID. It is a correctness problem, not just a performance one. In fact: Parquet's embedded schema protects correctness at any size — it is not just a scale optimisation.

**File names don't matter** — The contents of the file are what count; the name is just a label.

Pipelines find files by name patterns (`glob('sales_*.parquet')`). A file called `data_final_v2_really.parquet` breaks automation and makes it impossible to know what is inside. In fact: Predictable naming that encodes date, source, and version (e.g., `sales_2024-01_v2.parquet`) is a contract that lets downstream tools and humans know what to expect.


## In short

Use a columnar format (Parquet) with an embedded schema; partition on low-cardinality keys you actually filter by; and name files predictably. This keeps your data fast to read, correct in type, and safe to evolve — the bedrock of every data pipeline and every model you will later build.


## Where this is used

You may not write Parquet by hand, but you will rely on it everywhere. In **large language models**, training data is stored in Parquet files — for instance, it is common to see instruction-tuning datasets released as Parquet. When a developer loads such data, they use `pandas.read_parquet('path')`, which reads only the columns needed, and because the schema is embedded, the types are correct — a column of IDs stays as strings, not numbers. In **small language models running on devices** (like a phone keyboard), the model is quantized and stored in a different format, but the *training data* still comes from Parquet; the same column-pruning speedup means you can iterate faster on data preparation. In **retrieval-augmented generation**, you may store a knowledge base in Parquet to create embeddings later. The columnar layout lets you filter the data (like 'where language = "hi"') before sending it through an embedding model — reading only the text column, not the whole dataset. Partitioning by topic or date means you can rebuild only a slice of embeddings when new data arrives. In **agentic systems** that plan and call tools, a tool might need to fetch a data table; if that table is in Parquet, the agent can query it efficiently — for example, using DuckDB to run SQL on Parquet directly. Partitioning means a query like 'sales in March' touches only the March file, not a year of data. In **everyday applied ML** — fraud detection, recommendations, forecasting — the model's features are often prepared by a pipeline that outputs a Parquet file. A downstream training job reads that file with `pandas.read_parquet()`. If you use CSV instead, the reader must guess types, and a column like 'customer_id' with leading zeros loses them, silently corrupting the feature. With Parquet, the schema is explicit, so the model sees consistent types. What breaks when it is done badly? If you name files inconsistently (`data_2024-01.parquet`, `data_1_2024.csv`), a pipeline that globs for patterns may miss files or read the wrong one. If you partition on a high-cardinality column (like a unique customer ID), you create millions of tiny files — each one has overhead, and reading them is slower than one big file. If you do not plan for schema evolution, a new column added to one file and not another causes joins to fail. Developers handle this with tools like PySpark (`df.write.parquet('output')`) or DuckDB (`COPY tbl TO 'data.parquet' (FORMAT PARQUET)`), and they rely on the file format's embedded metadata to make the data self-describing.


---

Read this on the web: https://basavapurushottam.com/learn-ai/parquet-and-flat-file-hygiene
