# Protected Attributes

Characteristics such as race, gender, age or disability that law and ethics restrict from driving decisions.

**Rests on:** Regulatory Compliance Controls

*Trust, Governance and Ethics · Fairness and Bias*


## Why learn this

You deal with decisions that affect people’s lives in your professional work, whether hiring, lending, or approving services. Fairness auditing ensures those decisions do not unfairly penalize individuals based on who they are. If you ignore how hidden data characteristics can substitute for explicitly banned traits like race or gender, your systems may violate laws and ethical standards without you realizing it. Understanding this concept helps you spot these hidden risks early, preventing costly errors and protecting your organization’s reputation.


## The intuition

Imagine a loan application form. It asks for your name, age, gender, address, income, and a few more details. Some of those details, like race, gender, age, or disability, are what we call protected attributes. In many countries, a bank is legally barred from using them to decide whether to give you a loan. But the bank still needs to decide. So it looks at other things: your income, your savings, your job history, where you live, even your postcode. Here is the catch: your postcode is not your race, but in practice it can be strongly related to race. If the bank uses your postcode, it might be indirectly using race anyway. The whole problem of fairness in AI is that the rule 'do not use race, gender, age, or disability' is easy to write but hard to follow, because the model can find indirect signals that stand in for those characteristics. Removing the attribute from the form does not remove bias — the bias is hidden in the remaining data. That is the single most important thing to understand here.


## An analogy

Imagine a detective taking a witness statement. The witness says things like 'the robber was a tall man in a mask.' But the detective also notices the witness is nervous, maybe because the robber is still nearby. The witness's fear is not the robber's attribute, but it might be a signal of the robber's presence. The detective cannot directly use the robber's race because he does not know it, but he can use the witness's reaction. In the same way, a hiring algorithm is told not to consider gender, but it can consider whether a person took a career break, which is more common among women. The career break is a substitute marker for gender. The detective's mistake would be to say 'I never use race, I only used fear' — but the fear points at the same thing. The algorithm's mistake is to say 'I never use gender, I only used career breaks' — but the break points at the same thing. This analogy works until it breaks: a detective is trying to catch one criminal, while an AI system applies the same rule to thousands or millions of people. A detective can override a wrong signal per case, but the AI applies the marker automatically to everyone. That is why the problem is systematic, not occasional.


## Definition

A protected attribute is a characteristic such as race, gender, age, or disability that law and ethics restrict from driving decisions made by AI or human decision-makers.


## How it connects

This concept sits at the root of Fairness and Bias, which you will study next. Your library notes already name the neighbouring topics: black-box vs white-box assessment, perturbation robustness, sentiment and emotion bias, and trust rating scales. All of these use the idea of a protected attribute. When you assess a system black-box, you look for disparities across groups defined by protected attributes. When you perturb an input to test robustness, you are checking whether a change that should not matter — like swapping a name — changes the output, which would indicate the model is using a substitute marker for a protected attribute. Sentiment bias is just a specific case of protected attributes leaking into model scores. And every trust rating you eventually assign will hinge on how well the system handles protected attributes. So this is not an isolated topic; it is the lens through which all fairness questions are seen.


## The mathematics

The mathematics here is not a formula but a procedure for measuring disparity. You need to count how often a system makes a certain decision — approve a loan, hire a candidate, rate a credit score high — separately for each group defined by the protected attribute. For example, count the approval rate for men and for women. Then you subtract one rate from the other. That difference is a number you can report: 80% approval for men, 65% for women, a gap of 15 percentage points. This is arithmetic: counting and subtracting. The branch of mathematics is basic statistics — the part about averages and differences, not the advanced probability theory. The reason we use statistics is that a single decision may be fair while the pattern across many decisions is not. You need to aggregate many cases to see the disparity.

For a binary protected attribute (e.g., male vs female), the disparity is:

d = P(decision = positive | group = A) − P(decision = positive | group = B)

where P stands for 'the proportion of cases where', and the vertical bar means 'given that the person is in that group'. A common alternative is the ratio: r = P(positive | A) / P(positive | B), called the 'four-fifths rule' when r < 0.8 signals bias. Each symbol must be read as shorthand for the counting you just did by hand.


## A worked example

```
A bank logs decisions on 100 loan applications for one month. Each application records the applicant's gender (protected attribute in this hypothetical) and the outcome: approved or rejected. The data is already labelled, so all we do is count.

Step 1: Applications approved for male applicants: 40 out of 50.   # We count the male applicants and how many got approval.
Step 2: Approval rate for males = 40 ÷ 50 = 0.8 = 80%.   # Convert the count to a proportion by dividing approved by total.
Step 3: Applications approved for female applicants: 26 out of 50.   # Same counting for the other group.
Step 4: Approval rate for females = 26 ÷ 50 = 0.52 = 52%.   # Again a proportion.
Step 5: Disparity = 0.80 − 0.52 = 0.28 = 28 percentage points.   # Subtract the two rates to get the gap.

Result: The approval rate disparity is 28 percentage points, meaning the bank approves men at a substantially higher rate than women. This number is a starting point for investigation, not a verdict that the bank is discriminating — but it is the kind of number an auditor would flag.
```


## From scratch, in Python

```python
# Plain Python to compute a disparity (no libraries)

# Simulated data: each row is [gender, approved] where gender 1 = male, 0 = female
# approved is 1 if granted, 0 if not
applications = [
    [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1],
    [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1],
    [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1],
    [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1],
    [1, 0], [1, 0], [1, 0], [1, 0], [1, 0], [1, 0], [1, 0], [1, 0], [1, 0], [1, 0],
    # total 50 male: 40 approved, 10 rejected (the above list has 40 ones and 10 zeros)
    [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1],
    [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1],
    [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1],
    [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1],
    [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0],
    # total 50 female: 26 approved, 24 rejected (26 ones, 24 zeros)
]

# Counters for male and female groups
total_male = 0
total_female = 0
approved_male = 0
approved_female = 0

# Loop over each row: for each row, if gender is 1, increment male total;
# if approved is also 1, count as an approval for males. Similarly for females.
for row in applications:
    gender = row[0]
    approved = row[1]
    if gender == 1:
        total_male = total_male + 1
        if approved == 1:
            approved_male = approved_male + 1
    else:
        total_female = total_female + 1
        if approved == 1:
            approved_female = approved_female + 1

# Compute rates as proportions (float division)
rate_male = approved_male / total_male
rate_female = approved_female / total_female

# Disparity as absolute difference (percentage points)
disparity = rate_male - rate_female

# Print the results
print("Male approval rate:", rate_male)
print("Female approval rate:", rate_female)
print("Disparity (male minus female):", disparity)
```


## The same thing, with a framework

In practice, an auditor would use pandas, a Python library for tabular data. You would create a DataFrame (a table with named columns) and use the `groupby` method to calculate the mean of a column for each group of a protected attribute. For example:

```python
import pandas as pd

df = pd.DataFrame({'gender': [1, 1, 0, ...], 'approved': [1, 1, 1, ...]})
rates = df.groupby('gender')['approved'].mean()
disparity = rates[1] - rates[0]
```

Underneath, pandas is doing the same loop you wrote by hand: it goes row by row, groups the rows with the same gender, and calculates the average of the approval column for each group. The only difference is that pandas does this faster and with less typing.


## What the code does

This code does exactly what you did by hand in the mathematics section. First, we store the data as a list of two-element lists, which is just a way to keep two numbers together. A list is a Python object that holds other objects in order, separated by commas and surrounded by brackets. Here each inner list has two numbers: the first is gender (1 for male, 0 for female), the second is approval (1 for yes, 0 for no). Then we create four counters, initially zero. We loop over the outer list: the line `for row in applications:` means we take each inner list one at a time. Inside the loop, we break the row into its two parts using `gender = row[0]` and `approved = row[1]`. The `if` statements check whether the gender is 1 or 0, and within that check, whether approval is 1, incrementing the right counters. After the loop, we divide the counts to get rates, subtract to get disparity, and print everything. The only library is the standard `print` function.


## Common misconceptions

**Removing the attribute removes the bias** — If we just delete the column for race from the data, the model cannot be racist.

The model can still learn proxies, such as zip code, name, or even grocery purchase patterns, that correlate strongly with race. In fact: You must also remove or adjust for proxies — the measuring of disparity and auditing of features is what actually prevents bias.

**Protected attribute is always illegal to use** — Using gender or race in a model is always illegal.

Using the attribute to *measure* disparity is legal and necessary for auditing. It is using it to *decide* that is restricted. Also, which attributes are protected varies by jurisdiction and context (e.g., in India, caste is protected in some contexts). In fact: You may and should collect protected attribute data for audit purposes, with appropriate consent and privacy safeguards, to measure and correct disparities.

**Equal treatment means equal outcomes** — If the model does not look at gender, then it is fair to both men and women.

Even without using gender, the model may have different accuracy or different rates across groups because of underlying data correlations. Equal treatment (not using the attribute) is not the same as equal outcomes (e.g., equal approval rates). In fact: Fairness requires monitoring outcomes across groups and possibly adjusting the model, not just removing the attribute.

**The protected attribute is the one the model was trained on** — The only protected attribute is the one explicitly present in the training data.

Protected attributes are defined by law and ethics, not by what the model sees. Even if the model never sees 'gender', it may see proxies. In fact: You must consider any attribute that is a legal or ethical constraint, and test for proxies.

**A single disparity number means the model is biased** — If the approval rate gap is 20%, the model is definitely discriminating.

The disparity may be due to legitimate factors (e.g., historically women have lower income in some contexts, which is not protected in itself). The number is a starting point for investigation, not a verdict. In fact: Like your library notes say, measured disparity is a starting point for investigation, not a verdict. You need to examine the causal factors.


## In short

Protected attributes are characteristics like race, gender, age, or disability that law and ethics restrict from driving decisions. The core problem is that simply removing them from data does not remove bias, because models find proxies. To audit, you measure disparity across groups defined by protected attributes — compute the rate of a decision for each group and subtract. This disparity number is a starting point, not a verdict; you investigate further. Which attributes are protected varies by jurisdiction and context.


## Where this is used

In large language models (LLMs), protected attributes appear in two places. First, in training data: the text corpora contain statements about people with certain names, dialects, or descriptions, and the model learns statistical associations — e.g., that the name 'Priya' is more often associated with certain professions than 'Raj'. When you ask the model to write a recommendation letter or generate a resume summary, it may produce output that subtly steers based on those associations. To audit an LLM, you can generate counterfactual inputs: take a prompt, swap the name and gender pronouns (e.g., 'Alex' -> 'Priya'), and compare the outputs. If the sentiment or recommendation changes, the model is using the protected attribute as a proxy. This is a black-box test, as your library notes describe. Second, in model deployment: an LLM used for screening job applications can incorporate protected attributes indirectly through the style of writing (e.g., mentioning a women's college). There is no direct feature vector, but the embeddings — the numerical representations of text — encode those associations. Auditing typically involves probing with crafted prompts.

In small language models that run on devices (like a keyword spotter on a phone), protected attributes may seem less relevant because the model is predicting a phrase like 'Hey Siri'. But such models are trained on voice data, and accent, age, and gender affect pronunciation. A model that works well for young speakers may fail for older speakers, creating a disparity in the false rejection rate. The developer must collect speech data from a balanced set of speakers and test the false rejection rate across age groups. This is a protected attribute in the sense of disability too — speech impairments are a disability, and a model that does not work for slurred speech is excluding that group.

In retrieval-augmented generation (RAG), the system retrieves documents based on a query and then references them. If the embedding model used for retrieval has biases (e.g., it ranks male-authored documents higher for queries about 'leadership'), then the retrieved context will be skewed, and the downstream LLM answer will reflect that. Auditing here means building a test set of queries that are neutral with respect to gender, race, etc. and measuring whether the retrieved documents are balanced in authorship or content. You would compute the disparity metric we learned (e.g., fraction of retrieved documents authored by men vs women) and see if it deviates.

In agentic systems that plan and call tools, protected attributes can leak in when the agent makes decisions based on external databases. For instance, an agent that books a loan by querying a credit bureau: if the bureau's data includes postcodes and the agent uses a rule like 'avoid high-risk postcodes', that proxy may map onto race. The agent's plan generation could also be biased: if you ask it to 'write a cover letter', it might generate a letter for 'a woman with a career gap' differently than for a man. Auditing agentic systems is harder because the decision path is multi-step. You would instrument the system: log every tool call and the input/output of each step, then look for disparities in how often certain groups receive a negative outcome. The protected attribute may not be in the model's input at all, but the tool's result may encode it.

In everyday applied machine learning — recommendation, vision, forecasting, fraud detection, speech — protected attributes are routinely used (or should be) in fairness evaluation. For a recommender system, you compare top-1 hit rates across user groups defined by age or gender. For vision, you test face detection accuracy across skin tones (e.g., the Gender Shades study by Joy Buolamwini, which showed large disparities in error rates). For fraud detection, you audit whether the false positive rate differs by ethnicity; the model might be trained on historical data where certain groups committed more fraud, so it learns to flag them more. In practice, a developer writes a small evaluation script that groups the test set by the protected attribute (when allowed to have it for auditing) and computes the metric for each group, then reports the disparity. This is the same code we wrote above, using pandas or numpy.

The developer experience: when bias is detected, the fix is not trivial. You might remove the attribute, but you must also remove proxies, which requires feature selection and possibly causal analysis. The book 'Assessing, Explaining, and Rating AI Systems for Trust' in your library shows that causal graphs and backdoor adjustment allow you to estimate the protected attribute's true effect on the outcome, separating direct discrimination from legitimate factors. That is an advanced step; at the beginner level, you just measure the disparity.


---

Read this on the web: https://basavapurushottam.com/learn-ai/protected-attributes
