# Risk Tiers

Classifying use cases by potential harm, so evaluation depth, oversight and controls scale with the stakes.

**Rests on:** Cross-Validation Partitions

**Leads to:** Regulatory Compliance Controls, Test Planning

*Machine Learning · Model Evaluation*


## Why learn this

This unlocks Model Evaluation itself — the parent concept you have notes on. Once you can sort use cases into risk tiers, you know how much evaluation a model needs: a harmless fun app gets light checks, a medical diagnosis tool gets rigorous ones. That lets you read any evaluation guide, because every metric, test set, and monitoring plan assumes you understand the stakes. It also leads directly to your neighbouring notes: Unlearning Verification (higher tiers demand proof of forgetting, not just claims), plus Four-Axis Quality, Offline vs Online Evaluation, and Regression Gates — all of which only make sense once you know which tier they apply to.


## The intuition

Think of the stakes first, the maths later. Not every AI system is equally dangerous. A model that suggests a recipe for leftover rice is harmless if it's wrong; a model that tells a doctor a dosage could kill if it's wrong. Risk tiers are just a way of saying: the more harm a mistake can do, the more checking we do before we trust it. You already do this in daily life. You don't test a torch before buying it — you just check it works. But you'd test brakes on a car carefully before driving. Risk tiers formalise that common sense: they assign a level of danger to a use case, and that level decides how much evaluation, oversight, and control we apply. The tier isn't a score of how good the model is; it's a measure of how bad failure would be. The mapping from harm to controls should be written down before anything goes wrong, because during an incident panic replaces judgement. It's like deciding your emergency exit route before a fire, not during it.


## An analogy

Imagine you're hiring a security guard for a building. The guard is a model; the building is the use case. A small shop needs a guard who can open the door and smile — a light check. A bank vault needs a guard with background checks, training, and supervision. You wouldn't spend the same time verifying both. Risk tiers are the security levels you assign based on what's inside the building. The tiers map to concrete controls: how many tests the guard must pass, how often they're audited, what happens if they fail. They don't just give you a colour on a chart — they tell you exactly what to do. This analogy stops working when you realise a model isn't a person: a guard can explain their decision, a model usually can't. And a model's behaviour can change with new data, unlike a guard's personality. But the core idea holds: the cost of a mistake determines the effort you put into preventing it. And you must agree on the tier before the guard starts work, because during a burglary is no time to decide how careful you should have been.


## Definition

Risk tiers are a way of classifying AI use cases by the potential harm of a mistake, so that the depth of evaluation, the amount of human oversight, and the safety controls scale up with the stakes.


## How it connects

You're starting from zero, so I'll place this: Risk tiers aren't maths yet, but they're the foundation of Model Evaluation, which you have notes on. Think of Model Evaluation as 'measuring whether a model works' — and risk tiers tell you how carefully you must measure. A recipe suggester can get away with a rough accuracy check; a medical tool needs precise calibration and slice-based analysis. Your library also mentions Unlearning Verification, which is about proving a model has forgotten data. Risk tiers matter there because the verification required depends on the tier: a high-tier model must prove forgetting convincingly, a low-tier one can get away with a simpler test. So risk tiers tie into everything else — they're the lens through which you decide how much effort every other evaluation technique deserves.


## The mathematics

The branch here isn't calculus or algebra — it's basic arithmetic, plus a dash of logic. You don't need to plot graphs or solve equations. All you do is: list the possible failures, rate how severe each is, and combine those ratings into a single tier. It's like adding up scores, but the scores come from judgement, not from a formula. The 'mathematics' is really a way to be systematic: instead of guessing 'this feels dangerous', you break it into parts and add them up. The operations are: for each failure, assign a severity number (say 1 to 5) and a likelihood number (say 1 to 5), then multiply severity by likelihood to get a risk score. Then you sum those scores across all failures to get a total. Then you compare that total to thresholds — if it's above a certain number, you upgrade the tier. That's the whole maths. You also need to be able to compare numbers: is 15 higher than 10? Yes. That's it.

For each failure f: risk_f = severity_f × likelihood_f. Then total risk R = sum of all risk_f. Then compare R to thresholds T1, T2 (e.g., T1=10, T2=20) to assign tier: if R < T1, tier = low; if T1 ≤ R < T2, tier = medium; if R ≥ T2, tier = high.


## A worked example

```
Let's make it concrete. Imagine a chatbot that helps people book restaurant reservations. It can make mistakes. Let's list two possible failures and rate them. Failure 1: It books the wrong table. Severity: 3 out of 5 (mildly annoying, not dangerous). Likelihood: 4 out of 5 (it happens often). Failure 2: It double-books, causing a customer to be turned away. Severity: 4 out of 5 (more upsetting), Likelihood: 3 out of 5 (less frequent).

Step 1: Failure 1 risk = severity 3 × likelihood 4 = 12   # Multiply the severity by the likelihood to get the risk for that failure.
Step 2: Failure 2 risk = severity 4 × likelihood 3 = 12   # Same operation for the second failure.
Step 3: Total risk = 12 + 12 = 24   # Add the two risks together to get the overall risk.
Step 4: Compare to thresholds: Low < 10, Medium 10–20, High > 20   # These thresholds are arbitrary for this example, but they're set based on business judgement.
Step 5: 24 > 20, so the tier is High   # Since the total is above 20, we classify it as high risk.

Result: The chatbot is High risk, meaning we need detailed evaluations, human oversight, and rollback plans.
```


## From scratch, in Python

```python
def assign_tier(failures, threshold_low, threshold_high):
    total_risk = 0
    for severity, likelihood in failures:
        risk = severity * likelihood
        total_risk = total_risk + risk
    if total_risk < threshold_low:
        return 'low'
    elif total_risk < threshold_high:
        return 'medium'
    else:
        return 'high'

# Example usage:
failures = [(3, 4), (4, 3)]  # (severity, likelihood) for each failure
tier = assign_tier(failures, threshold_low=10, threshold_high=20)
print(tier)  # prints 'high'
```


## What the code does

This is plain Python. First, I define a function named assign_tier. It takes three inputs: failures (a list of pairs), and two thresholds (numbers). The word 'def' tells Python you're defining a function. Inside, I start a variable total_risk at 0. Then a for loop goes through each failure: the line 'for severity, likelihood in failures' takes each pair and splits it into two numbers, severity and likelihood. Then I multiply them to get risk, and add that to total_risk. After all failures are processed, I compare total_risk to the thresholds. If it's less than threshold_low, I return 'low'; otherwise, if it's less than threshold_high, I return 'medium'; else 'high'. The example uses failures as a list of two pairs: (3, 4) and (4, 3). The loop multiplies 3×4=12, then 4×3=12, adds to 24. 24 is not less than 10, and not less than 20, so it returns 'high'. The print statement displays that.


## Common misconceptions

**Risk tiers are about how accurate the model is** — If my model is 99% accurate, it's low risk, because it hardly ever makes a mistake.

Risk is about the harm from a single mistake, not how often it makes one. A model that's 99.999% accurate over a million predictions still makes 10 mistakes. If each mistake is a wrong medical dosage, that's dangerous. Accuracy reduces the likelihood but not the severity. In fact: Risk tiers combine both the severity of a mistake and the likelihood of it happening. A high-accuracy model can still be high risk if the consequences of any single mistake are severe.

**Risk tiers are just colours on a chart** — It's just a red, yellow, green label to show the boss. It doesn't change what I do.

If tiers don't change anything, they're decoration. The whole point is that they dictate the level of evaluation, oversight, and controls. A 'low' tier might get a quick sanity check; a 'high' tier demands extensive testing and human review. No mapping to actions means the tier is meaningless. In fact: Risk tiers are a checklist: each tier maps to specific actions like how many test cases to run, whether a human must approve, and what fallback plans to have.

**Risk tiers are decided during an incident** — We can figure out how dangerous it is after something goes wrong, so we'll decide then.

During an incident, people are panicking and judgement is poor. Decisions made under pressure are often inconsistent and too lenient. It's like writing a fire escape plan during the fire. In fact: Risk tiers must be agreed beforehand, with clear criteria, so that when a mistake happens, you just follow the plan without debate.

**Risk tiers are only for big tech companies** — My little project doesn't need this; it's for OpenAI or Google.

Even a small app that suggests movies can have a failure that hurts someone, like recommending something inappropriate to a child. And regulatory frameworks are starting to require this classification for any AI system. A small startup can be sued for negligence if they didn't classify their risk. In fact: Any AI system, regardless of size, benefits from risk tiers. It's about due diligence, not company size.

**Risk tiers are static once assigned** — Once I've decided it's low risk, I can forget about it.

The use case can change. If you start using the model in a different domain, or if the data changes, the risk could go up. A harmless chatbot that suddenly gets used for customer service for a bank is now riskier. In fact: Risk tiers should be reviewed regularly, especially when the model is used in a new way or when significant updates are made.


## In short

Risk tiers are a simple arithmetic system: multiply the severity of each failure by its likelihood, add them up, and compare to thresholds to get a tier. The tier then dictates how much evaluation and control you apply. The key is to set tiers before anything goes wrong, and to map them to concrete actions, not just labels.


## Where this is used

This concept is used everywhere in AI, though it's often invisible. In large language models (like ChatGPT), developers don't just release the raw model to everyone. They classify use cases: a general chat assistant might be medium risk, but a tool that gives legal advice or medical information is high risk. The developers then apply different levels of content filtering, human review, and evaluation. For example, OpenAI has usage policies that restrict certain high-risk domains. The mechanism: they set up tiers, and for high-risk uses, they require additional safety measures like fine-tuning to avoid harmful outputs, and they run intensive evaluations before release. In small models running on devices (like a smartphone's keyboard autocomplete), the risk is lower because the model's mistakes are less consequential. But still, they classify: a model that suggests emojis is low risk, but a model that predicts what you're typing for a banking app is medium risk because typos in a bank account number could be serious. Developers write code that checks the tier in their evaluation pipeline to decide how many test cases to run. In retrieval-augmented generation (RAG), where a model pulls information from a database to answer questions, risk tiers are crucial. If the RAG system answers medical questions, the risk is high; a wrong answer could mislead a patient. Developers then evaluate not just the model's answer, but also the quality of the retrieved information, and they add warnings to the interface. In agentic systems that plan and call tools (like an AI that sends emails or makes purchases), risk tiers are essential. A system that just checks your calendar is low risk; a system that can spend money is high risk. Developers implement tier-based controls: for high risk, they require human approval for every action. They write code that checks the tier and blocks certain actions if it's above a threshold. In everyday applied ML (like recommendation engines or fraud detection), risk tiers guide the evaluation metrics. A movie recommendation that's wrong is harmless — low tier, so simple accuracy is fine. A fraud detection model that wrongly blocks a transaction is annoying; a model that misses a real fraud is terrible. So they set tiers, and for high tiers they tune precision (avoid false positives) versus recall (catch all fraud) carefully. What a developer actually writes: they often start with a spreadsheet of use cases, rate each failure with severity and likelihood, sum the risks, and then use that to decide what tests to run. In code, they might have a function like 'get_evaluation_depth(tier)' that returns how many test examples, what metrics, and what oversight is needed. If done badly, they might assign tiers too late, after a mistake happens, or they might ignore tiers and evaluate everything the same way, leading to either wasted effort or dangerous oversights. The key is to decide tiers before building the model, and to update them as the model's usage evolves.


---

Read this on the web: https://basavapurushottam.com/learn-ai/risk-tiers
