# Unicode and Encoding Pitfalls

Text failures caused by encoding mismatches, normalisation differences and the gap between bytes, code points and graphemes.

**Rests on:** Data Transformation

*Data Engineering and Analytics · Data Quality*


## Why learn this

Before you can clean data, you must be able to read it accurately. Unicode and encoding pitfalls are silent killers of text data — they corrupt data without raising any error, so you only discover the problem after your analysis has produced wrong results. This section teaches you to see the three layers of text representation: the raw bits stored on disk, the unique symbols assigned by a standard system, and the visual characters people actually read. By understanding these layers, you will learn to spot and fix encoding mismatches and understand why two strings that look identical on screen are not equal in computer memory. With this knowledge, you can debug 'mysterious garbled character' errors, solve 'why is my foreign language text unreadable?' puzzles, and fix 'I swear these names are the same but the database says they differ' conflicts. It allows you to write data validation tests that catch these problems early and helps you understand what developers mean when they say 'the encoding is wrong', ensuring your downstream tasks that touch text — which is almost all of them — produce correct results.


## The intuition

Think of a text message. You type 'hello' on your phone. That message travels through the internet as a series of 1s and 0s. When it arrives, your friend's phone transforms those 1s and 0s back into the letters 'hello'. The 1s and 0s are the bytes, the raw digital units used for storage and transmission. The letters are the characters. But there is a third level: what you actually see on the screen. Consider the word 'café'. It has four visible characters: c, a, f, and é. However, in computer memory, this word can be written in two different ways. In one way, 'é' is stored as a single symbol (a single code point). In another way, it is stored as 'e' followed by a tiny accent mark that combines with the 'e' to form the visual shape (two separate code points, where the second is a combining mark — a character that modifies the one before it). Both look exactly the same on the screen, but they are stored as different sequences of numbers. Now imagine you have two lists of customer names. One list uses the single-symbol version of 'é', and the other uses the two-part version. When you try to merge them using a standard 'match' command, the names don't match — even though a human sees 'café' in both cases. This is the core trap: bytes, unique symbols (code points), and visible characters (graphemes) are three different ways of counting or representing the same text. The same visible string can be stored in multiple valid ways, and if your software doesn't know which way to expect, it gets surprised by mismatches.


## An analogy

Imagine you have a bookshelf full of novels. Each novel is written on loose pages. Think of the book as the 'grapheme' — the final visual story you read. Each page is a 'code point' — a single unit of meaning assigned by the publishing system. Now, imagine that some editions of a novel print certain letters using special ink blends. For example, one edition prints 'café' using four distinct pages: c, a, f, and a pre-mixed page for 'é'. Another edition prints it using five pages: c, a, f, e, and a separate thin page containing just an accent mark that sits visually on top of the 'e' page. Both editions look identical when held up to the light, but the physical composition is different. If you try to sort the books by counting their pages, the two editions will have different counts (4 vs 5). This represents an encoding mismatch: silently changing how data is composed breaks logical operations like sorting or matching, even though the visual content appears unchanged. Where this analogy breaks down is that in a real bookshelf, you could physically feel the difference in page count if you were careful. In a computer, you cannot see the difference between the two editions on screen — both look exactly the same. You need specific tools to peek inside and inspect the individual pages (code points) underneath.


## Definition

Unicode and encoding pitfalls are text errors caused by mismatches between how text is stored as raw bits, how it is represented as unique symbols (code points), and how those symbols visually combine into characters (graphemes), leading to silent data corruption because identical-looking strings may have different underlying digital structures.


## How it connects

This concept sits inside your Data Quality notes. Data Quality is about whether data is accurate, complete, and fit for purpose. Encoding problems are a classic silent degradation: the pipeline succeeds without throwing an error but produces wrong data. This connects directly to Data Transformation because converting text between formats (like changing from one encoding style to another) is a transformation that must produce identical results every time. It also connects to Data Validation Constraints because you can write rules that catch these issues, such as requiring all text fields to use a specific standard format or ensuring no unexpected combining marks are present.


## The mathematics

This concept does not rest on algebra or calculus. It rests on counting and on set membership. You need to know how to count the number of items in a sequence, and you need to know what it means for two sequences to be equal. The mathematics here is about equality: when are two strings the same? In mathematics, equality is strict: two things are equal only if they have exactly the same components in the same order. For text, we have three levels of counting. The first is bytes: the raw numbers that the computer stores. The second is code points: the abstract characters from the Unicode standard (like 'A' or 'é' or '₹'). The third is graphemes: the visible symbols a human sees (like 'é' but also a combined 'e' plus accent). The trap is that the same grapheme can be made from different code points, and the same code point can be stored as different bytes depending on the encoding. So two strings can look the same to a human but be different sequences of code points, or different sequences of bytes. The mathematical operation you need is a comparison: are these two sequences identical? And the pitfall is that the comparison might be done at the wrong level. If you compare bytes and they are different, but the code points are the same (that is, two different encodings of the same code point), then you get a false mismatch. If you compare code points and they are different, but the graphemes are the same (normalisation difference), you get a false mismatch. The only way to get a correct result is to compare at the same level, after normalising.

Let S be a string. We can represent it as:
- Bytes: B(S) — a sequence of numbers.
- Code points: C(S) — a sequence of Unicode code points.
- Graphemes: G(S) — a sequence of visible symbols.
Equality comparison: S1 == S2 is true only if the sequences at the same level are identical. Comparing at different levels (e.g., bytes of S1 with code points of S2) is meaningless. Normalisation (NFC or NFD) is a function N that maps a string to a canonical form, so that N(S1) == N(S2) when the graphemes are the same.


## A worked example

```
Consider the string 'café'. We will write it in two ways: (1) with a single character 'é' (code point U+00E9), and (2) with an 'e' followed by a combining accent (code points U+0065 and U+0301). Both look identical on the screen. We will also use two encodings: UTF-8 and Latin-1 (also called ISO-8859-1). We will count the bytes for each.

Step 1: Version 1 (single é): code points are 'c', 'a', 'f', 'é' → 4 code points.   # Counting the characters (code points).
Step 2: Version 1 in UTF-8: 'c' is 1 byte, 'a' is 1 byte, 'f' is 1 byte, 'é' is 2 bytes (C3 A9) → total 5 bytes.   # UTF-8 uses 1 byte for ASCII, 2 bytes for common accented Latin characters.
Step 3: Version 1 in Latin-1: 'é' is 1 byte (E9) → total 4 bytes.   # Latin-1 uses 1 byte per code point for the first 256 characters.
Step 4: Version 2 (e + combining accent): code points are 'c', 'a', 'f', 'e', U+0301 → 5 code points.   # The accent is a separate code point.
Step 5: Version 2 in UTF-8: 'e' is 1 byte, U+0301 is 2 bytes (CC 81) → total 5 bytes.   # ASCII e is 1 byte, the combining accent is 2 bytes.
Step 6: Version 2 in Latin-1: U+0301 is not in Latin-1, so it cannot be represented. If you force it, you get an error or a '?' placeholder.   # Latin-1 has no combining characters.

Result: The same visible 'café' can be stored as 4, 5, or an error depending on the encoding and normalisation. Two strings that look identical may have different byte sequences, so a byte-level comparison will say they are different, even though a human sees the same word.
```


## From scratch, in Python

```python
# Let's explore bytes, code points, and normalisation in plain Python.

# First, a string literal.
s1 = "café"  # This is version 1 (single é)

# To see the code points, use the built-in ord() function.
# ord() takes a single character and returns its Unicode code point number.
# It is a simple lookup.

# Let's encode the string into bytes using different encodings.
# The encode() method returns a bytes object, which is a sequence of integers.

# UTF-8 encoding
bytes_utf8 = s1.encode("utf-8")
print("s1 in UTF-8:", bytes_utf8)  # b'caf\xc3\xa9'

# Latin-1 (also called ISO-8859-1) encoding
bytes_latin1 = s1.encode("latin-1")
print("s1 in Latin-1:", bytes_latin1)  # b'caf\xe9'

# ASCII encoding will fail because é is not in ASCII.
try:
    s1.encode("ascii")
except UnicodeEncodeError as e:
    print("ASCII error:", e)

# Now let's construct the second version: 'e' plus a combining accent.
# The combining accent is code point U+0301, which we can write as "\u0301" in Python.
s2 = "cafe\u0301"  # This is 'e' followed by the combining acute accent

# Print both strings. They look the same!
print("s1:", s1)
print("s2:", s2)

# But are they equal at the byte level? Let's encode s2 in UTF-8.
bytes_utf8_s2 = s2.encode("utf-8")
print("s2 in UTF-8:", bytes_utf8_s2)  # b'cafe\xcc\x81'

# They are different byte sequences.
print("Byte sequences equal?", bytes_utf8 == bytes_utf8_s2)  # False

# At the code point level, they are also different:
# s1: c, a, f, é (U+00E9)
# s2: c, a, f, e, U+0301
# Let's print the code points as an explicit list.
print("Code points of s1:", [hex(ord(ch)) for ch in s1])
print("Code points of s2:", [hex(ord(ch)) for ch in s2])

# But a human sees the same characters. To compare them fairly, we normalise.
# Python has a module called unicodedata for this.
import unicodedata

# NFC (Normalization Form C) composes the characters: e + accent becomes é.
norm_s2 = unicodedata.normalize("NFC", s2)
print("Normalised s2:", norm_s2)  # café

# Now, the code points are the same as s1.
print("Code points of norm_s2:", [hex(ord(ch)) for ch in norm_s2])

# And the byte sequences are the same as s1's UTF-8.
print("Byte sequences equal after NFC?", s1.encode("utf-8") == norm_s2.encode("utf-8"))  # True

# Another useful normalisation is NFD, which decomposes: é becomes e + accent.
norm_s1_nfd = unicodedata.normalize("NFD", s1)
print("NFD of s1:", norm_s1_nfd)  # cafe\u0301
print("Code points of NFD of s1:", [hex(ord(ch)) for ch in norm_s1_nfd])

```


## What the code does

We start with a string literal 'café'. You don't need to know about lists or functions to follow this. The key idea: a string is like a list of characters, but you can ask it to give you the bytes. The encode() method turns the abstract characters into a sequence of numbers — the bytes. We do this with two encodings: UTF-8 and Latin-1. The same character 'é' uses 2 bytes in UTF-8 but 1 byte in Latin-1. Then we construct the second version of the same visible word: 'e' followed by a combining accent (written with the escape sequence \u0301 in Python). When we print s1 and s2, they look identical on screen. But when we compare their bytes, they are different: s1.encode('utf-8') gives b'caf\xc3\xa9', and s2.encode('utf-8') gives b'cafe\xcc\x81'. That's the crux: identical-looking strings can have different bytes. To bring them to a common form, we use unicodedata.normalize('NFC', s2), which composes the 'e' and the accent into a single 'é'. After that, both strings have the same code points and the same bytes, so a library that compares bytes will now say they are equal.


## Common misconceptions

**Encoding is just Unicode** — I thought Unicode is the only encoding, and everything in Python is Unicode, so I don't need to worry.

Unicode is a standard that defines code points, not a specific byte encoding. UTF-8, UTF-16, and UTF-32 are different encodings of Unicode. Also, there are many other encodings (Latin-1, Shift-JIS) that are not Unicode. When reading a file or API response, you must know which encoding was used to write it. In fact: Unicode is a mapping from characters to code points. An encoding like UTF-8 defines how those code points are stored as bytes. You must always specify the encoding when reading or writing text.

**If it prints correctly, the data is fine** — I check the data by printing it and it looks correct, so there's no problem.

Two different byte sequences can print the same. The screen shows graphemes, not bytes. You need to inspect the actual code points and bytes to be sure. In fact: Printing only shows the grapheme layer. To verify encoding, inspect `len(s.encode('utf-8'))` and the list of code points.

**All text is ASCII** — I just use English text, so I don't have to worry about encodings.

Even English text can have non-ASCII characters like curly quotes, em-dashes, or currency symbols like ₹. Also, international data will inevitably have non-ASCII. In fact: Assume any text may contain non-ASCII. Normalise and encode explicitly.

**Normalisation is only about accents** — Normalisation only affects accented characters, which I don't have in my data.

Normalisation also affects other character classes: ligatures (like 'ﬁ'), Chinese characters, and some punctuation. Also, some characters have multiple representations beyond combining marks. In fact: Use normalisation on all text to ensure consistency, regardless of the language.

**Decoding errors cause crashes, so I'd notice them** — If the encoding is wrong, I'll get an error message, so I'll know immediately.

Some encodings are forgiving — they can decode any byte to some character, but produce mojibake (garbage). The pipeline succeeds, and the error is discovered only when analysis gives nonsensical results. In fact: Watch for silent corruption. Use explicit encoding and test for expected characters.


## In short

Text has three layers: bytes, code points, and graphemes. A visible string can be stored in multiple ways. Encoding mismatches and normalisation differences cause silent corruption and failed comparisons. Normalise and encode explicitly, and test your data for these issues.


## Where this is used

This concept is used everywhere text data flows through an AI system. In **large language models**, the model's tokenizer splits text into tokens. A tokenizer operates on code points or bytes, not graphemes. If your input text has a normalisation difference (e.g., one 'café' is NFC and another is NFD), the tokenizer may produce different tokens, and the model will treat them as different words. This can degrade the model's understanding of synonyms or named entities. Developers often normalise all input to NFC before sending it to the model. In **small language models that run on a phone**, memory is tight, and the model's vocabulary is limited. The tokenizer must handle the byte representation efficiently. If the input is not in UTF-8 (e.g., it is in Latin-1 from an old system), the tokenizer may produce garbage tokens, and the model will output nonsense. In **retrieval-augmented generation**, text is embedded into vectors. The embedding model is trained on specific byte or code point sequences. If two documents have the same text but different normalisation forms, their embeddings will be different, and a similarity search may fail to retrieve one of them. Developers must ensure that the text going into the embedding model is normalised consistently. In **agentic systems** that call tools and act in loops, the agent may receive text from an external API that uses a different encoding (e.g., a legacy system that uses Shift-JIS for Japanese). If the agent decodes it with the wrong encoding, it will misread the content and make wrong decisions. The developer must include an encoding detection or conversion step in the agent's pipeline. In **everyday applied machine learning** — like building a recommendation system from customer reviews, a vision model that reads image captions, fraud detection on transaction descriptions, or speech recognition on audio transcripts — the same issue appears. Two records that should be identical (e.g., a product name 'café' from two different databases) may not match because of a normalisation difference. This causes duplicate records in training data, inflating counts and leaking between train and test splits. The developer writes a validation test that asserts all text fields are in a canonical normalisation form and a supported encoding, using a library like `unicodedata` or a data validation tool like Great Expectations. The test runs as part of the data pipeline and fails loudly if the encoding is wrong, instead of allowing silent corruption.


---

Read this on the web: https://basavapurushottam.com/learn-ai/unicode-and-encoding-pitfalls
