# Canary Releases

Routing a small traffic share to a new version, comparing metrics, and expanding only if it holds up.

**Rests on:** Benchmarking

*MLOps and Platform · MLOps*


## Why learn this

You've mastered the basics of building and training models. Now you're stepping into how those models actually reach users. Canary releases are the safety net that lets you ship a new model or feature without risking the whole system. Master this and you unlock the tools of MLOps: staged rollouts, automatic rollback, and the confidence to deploy continuously. This is the difference between a model that works in a notebook and a model that works in production.


## The intuition

Imagine you're a chef who has invented a new recipe. You could serve it to every customer in the restaurant at once, but if they hate it, you've ruined everyone's evening. Instead, you try it on just one table. You watch them eat. Do they smile? Do they ask for seconds? If they love it, you slowly offer it to more tables. If they push it away, you pull it from the menu before anyone else is disappointed. A canary release works exactly like that. You have a new version of your AI system, maybe a new recommendation model or a new way of detecting fraud. You don't switch everyone over at once. You route a tiny fraction of your real users—say, 5%—to the new version. Then you watch the metrics carefully. Are those 5% of users getting good results? Is the system responding as fast as before? Are there more errors? If everything looks good, you increase the percentage. You might go from 5% to 20%, then to 50%, and finally to 100%. If at any point the metrics get worse, you automatically switch those users back to the old version. The new version is the canary, and it's our early warning system. The whole point is to limit the blast radius—if the new version is bad, only a few people experience it, and you can quickly roll back.


## An analogy

Think of a new employee starting at a company. On their first day, you don't give them access to the entire customer database or the ability to approve million-rupee transactions. You give them a few small tasks. You watch how they perform. Do they make mistakes? Are they faster than the previous person? If they do well, you gradually give them more responsibility. Over a few weeks, they earn the trust to handle the big accounts. This is the same principle as a canary release. The new version of your model starts with a small amount of 'responsibility'—a small percentage of traffic. It has to prove itself before it gets the full job. The analogy breaks down a little, because a human employee can be interviewed and has references, but a new model version is a total unknown. We can only judge it by its actions on real data. Also, an employee learns and improves, but a model is frozen at deployment. If it's bad, it's bad, and we need to remove it quickly. The canary release is designed for that: it gives us a way to get it out of the kitchen without a scene.


## Definition

A canary release is a deployment strategy where a small percentage of live traffic is routed to a new version of a system, its performance is compared against the current version, and the rollout is expanded or rolled back based on the observed metrics.


## How it connects

You already have notes on Model Deployment and CI-CD. Canary releases fit right in between them: after you've packaged your model and set up the pipeline to deploy it, a canary release is the safe way to actually push it to users. It's closely tied to your notes on Release and Rollback Safety. In fact, a canary release is one of the primary tools for making releases reversible. The whole idea is to have a rehearsed path back to the old version if things go wrong. It also connects to your notes on Model Registry and Versioning—you can't do a canary release if you can't instantly point back to the previous version. And it's the practical answer to the question of how to safely put a new model from your Model Registry into the model serving API. If the canary doesn't work, you'll rely on your Drift Detection notes to help figure out why. Canary releases are also a key part of the broader MLOps practice of closing the loop from production feedback back into the next model.


## The mathematics

The mathematics of canary releases isn't about calculus or algebra. It's about arithmetic and simple probability. The core question is: 'Are the metrics we're seeing from the new version worse than the metrics from the old version, or is the difference just due to random chance?' This is the branch of mathematics called statistics, and specifically hypothesis testing. But we won't use any heavy notation. The idea is simple: if you have a small sample, you can't be sure what you're seeing is real. For example, if you route 5% of traffic to the new version, and 1% of users normally encounter an error, then with 100 users that's just 1 error out of 5 for the new version. If you see 2 errors, is that a real problem? Maybe, but it could be just bad luck. The larger the sample of users, the more confident you can be. We need enough traffic to be 'statistically meaningful'. This is the key point from your notes: 'Comparison needs enough traffic to be statistically meaningful.' It's not just about the number of users; it's about the rate of errors or other metrics. If the new version is supposed to be better at, say, predicting what product a user will buy, then we need to compare the average accuracy or click-through rate to the old version. The math is just about calculating rates and seeing if the difference is bigger than what you'd expect from random chance.

There's no single formula for canary releases, but the key statistical concept is the sample proportion.

The error rate (or any other metric) for a group is:

p = (number of events) / (total number of observations)

For comparing two rates, you can look at the difference: Δ = p_new - p_old

You need enough observations (n) so that the difference isn't just noise. A rough rule of thumb is that the standard error of the difference is sqrt(p(1-p) / n) (a more advanced statistics topic).


## A worked example

```
Suppose you have 10,000 users per hour. Your current version has an error rate of 2% (that is, 2 out of every 100 requests result in an error). You deploy a new version as a canary, routing 5% of your traffic (500 users) to it. After one hour, you look at the error rates.

Step 1: Old version: 95% of 10,000 = 9,500 users.
Errors (old) = 2% of 9,500 = 0.02 × 9,500 = 190 errors.   # Calculating the number of errors on the old version, assuming it behaves as usual.
Step 2: New version: 5% of 10,000 = 500 users.
Errors (new) = ? Let's say you see 15 errors.
Error rate (new) = 15 / 500 = 3%.   # The new version shows a higher error rate than the old one. Is 3% vs 2% a real difference or just due to the small sample size?
Step 3: With 500 users, the expected number of errors at 2% is 10. You saw 15. The difference is 5.   # This difference could happen by chance. To be more confident, you might wait for more traffic, say 5,000 users, and see if the 3% rate holds.
Step 4: If you had routed 50% (5,000 users) and the new version still had 3% errors, that would be 5,000×0.03 = 150 errors. The old version would have 5,000×0.02 = 100 errors. The difference is 50.   # A larger sample gives you more confidence that the 1% difference is real, not just a fluke.

Result: You decide to keep the canary at 5% for now. After collecting 10,000 users on the new version, you see 290 errors, which is 2.9%. The old version on that same traffic would have had 200 errors (2%). The difference is 90 errors. This is a significant increase. You trigger the automatic rollback and route all traffic back to the old version.
```


## From scratch, in Python

```python
# Canary release logic (simplified)

import random

# A list of user IDs (in reality, this would be a stream)
user_ids = ["user_001", "user_002", "user_003", "user_004", "user_005"]

# The canary percentage
canary_percent = 10  # 10%

# Function to decide which version to send a user to
def route_user(user_id):
    # Use a hash to deterministically route the same user to the same version
    user_hash = hash(user_id)
    if (user_hash % 100) < canary_percent:
        return "new"
    else:
        return "old"

# Simulate a few requests
for user in user_ids:
    version = route_user(user)
    print(f"{user} -> {version}")
```


## The same thing, with a framework

```python
In practice, you wouldn't write this routing logic yourself. You'd use a load balancer or an API gateway like nginx, AWS ALB, or Envoy. They have built-in features for percentage-based routing. For example, in Kubernetes, you can use Argo Rollouts or Flagger to manage canary releases for your ML model containers.
```


## What the code does

First, we import the `random` module, but we actually don't use it because we use `hash()` for deterministic routing. A list `user_ids` holds a few sample user IDs. We set `canary_percent` to 10. The `route_user` function takes a `user_id`, hashes it, and then uses the modulo operator `%` to get a number between 0 and 99. If that number is less than `canary_percent`, we send the user to the "new" version; otherwise, to the "old" version. This ensures that the same user always goes to the same version (as long as the canary percent doesn't change), which is important for consistency. The loop iterates over each user and prints which version they go to. This is the core idea behind canary releases: route a small, deterministic share of traffic to the new version.


## Common misconceptions

**Canary release means just deploying to a test server** — I'll just deploy the new model to a staging server and test it there, that's a canary release.

A staging server doesn't have real traffic. It's an isolated test environment. Canary releases are about routing a small percentage of real, live traffic to the new version. This is what makes the test realistic—you see real user behaviour, real system load, and real interactions. In fact: A canary release is about exposing the new version to a small fraction of real users, not to a test environment.

**Canary release is the same as gradual rollout** — Gradually increasing the percentage from 0% to 100% is all there is to a canary release.

A gradual rollout is just the mechanism. The true essence of a canary release is the comparison. You are comparing the performance of the new version against the old version. If the new version performs worse, you need to be able to automatically roll back. A gradual rollout without that comparison and rollback mechanism is just an empty ramp-up. In fact: A canary release is a controlled experiment: compare the new version against the old, and use the results to drive the rollout or the rollback.

**Canary releases are only for software bugs, not for AI models** — Canary releases are for finding coding errors. My model is good because I tested it offline. I can just deploy it directly.

ML models have their own set of problems: concept drift, data skew, and latent bugs. Offline evaluation can't capture the real-world distribution of data. A canary release is your best defence against these issues. It's specifically for testing the model's behaviour in production, under real data. In fact: Canary releases are a standard part of ML deployment to test for model-specific issues like data skew and concept drift.

**The canary should be a large percentage to get meaningful results quickly** — I need to send 70% of my traffic to the new version to see if it's better quickly.

This defeats the purpose. The entire idea is to limit the blast radius. If the new version is bad, you don't want 70% of your users to experience it. A small percentage (e.g., 5%) is often enough to get statistically meaningful results, especially if you have high traffic and a clear metric. In fact: Start small (5% or 10%). The key is to have enough traffic to see a difference, not to expose everyone to a bad version.

**Canary releases are only for the model itself** — The model is the only thing that changes, so I only need to canary the model container.

The release unit is the whole prediction contract. In MLOps, a new model often comes with changes to the feature engineering code, the schema, and the thresholds. Changing any of these can cause issues. You need to treat the whole package as a single unit for the canary release. In fact: Canary testing should include the model and all its associated dependencies: feature transformation code, serving infrastructure, and configuration.


## In short

A canary release is a safety device for deployment. Route a small share of real traffic to the new version, compare its metrics to the current version, and expand or roll back automatically. The key is that the comparison and rollback are automatic, so you never expose the whole user base to an unproven version. You need enough traffic to be statistically meaningful, and you need a defined regression gate that triggers a rollback. This is the core of safe MLOps.


## Where this is used

Canary releases are used throughout the AI industry, from large tech companies to small startups. Here's how:

- **Large Language Models (LLMs):** When OpenAI or Google updates GPT or Gemini, they don't instantly switch all users. They likely use a canary release. For a small percentage of requests, they might route to the new model, while the rest continue on the old one. They compare metrics like response quality (as measured by automated evals), latency, and safety violations. If the new model is better, the canary percentage increases. This is critical because LLMs are expensive to run and a regression could be catastrophic. The 'Practical LLM Evaluation for Production Systems' book emphasises evaluation across quality, safety, cost, and reliability—canary releases are the mechanism to test those dimensions in production.
- **Small language models on devices:** When a company like Apple or Google updates a small language model that runs on your phone, they do a canary release through the app store. A fraction of users get the new version of the app with the new model, and they monitor crash reports and user ratings. If the app crashes more or users complain about bad autocomplete, they can push a hotfix. Since the model runs on the device, the 'rollback' is a request to update the app again—but the damage of a bad model on a device is limited to that device.
- **Retrieval-Augmented Generation (RAG):** RAG systems often have multiple components: a retriever, a re-ranker, and an LLM. You can canary-release new versions of each. For example, you might have a new embedding model that changes the retrieval step. You can route a small percentage of queries to the new embedding model and measure whether the responses are more relevant. This is easier than a full LLM rollout because the cost is lower and the impact is more contained.
- **Agentic systems:** Agentic systems (AI agents that perform tasks) are more unpredictable than a simple model. A canary release is essential. By broadcasting a small percentage of tasks to a new agent version, you can monitor how well it completes the tasks, whether it calls tools correctly, and whether it causes any unintended side effects. The book points out that tracing (using MLflow Tracing) is key here—you need to see the full execution of a request (prompts, tool calls, outputs) to diagnose issues. Canary releases allow you to do that at low volume.
- **Everyday applied ML:** Canary releases are a standard practice for any ML system in production that makes recommendations, forecasts, or decisions. For example, a bank using an ML model to approve credit cards will canary-release a new model to a small percentage of loan applicants. They compare the default rate and the approval rate against the old model. If the default rate goes up, they roll back. An e-commerce company will canary-release a new recommendation model, watching for changes in click-through rate and conversion. This is also used in fraud detection: a new model might be tested on a small percentage of transactions, and if the false positive rate (blocking legitimate transactions) increases, it's rolled back immediately.
- **What a developer actually writes:** In practice, a developer typically doesn't write the canary logic from scratch. They use infrastructure tools like:
    - **Kubernetes + Argo Rollouts** or **Flagger**: These tools manage the traffic percentage and automatically scale up or down. They can also run a smoke test (a simple 'is the new version working?' test) before shifting traffic.
    - **Cloud providers' load balancers**: AWS ALB, GCP Load Balancing, and Azure Front Door all support weighted target groups. You can set a 5% weight to a new version.
    - **Feature flags**: Tools like LaunchDarkly or Split let you turn on a new version for a small percentage of users. This is common for canary releases of ML features that are part of a larger application.
- **What breaks when it's done badly:** If you do canary releases poorly, the blast radius is not limited. If you route 50% of traffic to a new model that is broken, you could be serving millions of users garbage. If you don't have automatic rollback on metric breach, you have to manually intervene, and by the time you do, you've already produced months of bad predictions. The book 'Practical LLM Evaluation' stresses that you need to define regression gates beforehand: what metric drop should trigger a rollback? Without these, your canary is just flying blind.


---

Read this on the web: https://basavapurushottam.com/learn-ai/canary-releases
