# Container Orchestration Basics

Managing container lifecycle across restarts and hosts — scheduling, health checks, restart policies and service discovery.

**Rests on:** Docker Compose

**Leads to:** Backup and Hardware Failure

*MLOps and Platform · Self-Hosting*


## Why learn this

You already run Docker and Docker Compose — you can start a multi-service stack on one machine with one command. But what happens when a container crashes at 3 a.m.? Compose will not bring it back on its own; that requires restart policies and health checks, the first pieces of orchestration you will see here. Later, when you self-host more than one host — a second machine, a VPS, a spare Raspberry Pi — you will need scheduling (which host runs what), service discovery (how components find each other without hardcoded IPs), and scaling (more copies when load rises). That is exactly what orchestration provides. Mastering this concept lets you go on to Docker Swarm, Kubernetes, and the operational side of self-hosting large AI systems — and it is the bridge from 'I can run containers' to 'my services stay up on their own.'


## The intuition

Think of a container as a single process — a web server, a database, a model server — packed with everything it needs to run. Docker Compose lets you define several containers and run them together on one machine. Container orchestration is the next step: it manages the lifecycle of those containers across restarts and across multiple hosts. Orchestration handles four main jobs. First, scheduling: deciding which host runs which container, and moving containers if a host dies. Second, health checks: asking each container, 'Are you still alive and doing useful work?' and acting on the answer. Third, restart policies: what to do when a container exits — restart it, back off, or give up. Fourth, service discovery: a way for containers to find each other by name, not by IP address, so you can move them around without breaking connections. The magic is that a service becomes self-healing — if something dies, the orchestrator notices and repairs it automatically.


## An analogy

Imagine you run a small restaurant. Each dish is a container—a self-contained recipe that can be cooked anywhere there is a kitchen. Docker Compose is your home kitchen: you know exactly where everything is, and you can cook one meal for a few friends. But now you want to open a chain with a dozen kitchens across the city. You cannot stand over every stove. You hire a manager (the orchestrator). The manager's job: assign each dish to a kitchen (scheduling), check that each dish is being cooked correctly (health checks), and if a chef walks out (container crash), find a replacement and restart that dish (restart policy). Also, the kitchens need to talk to each other—the sauce station needs to know where the main course is being cooked. They use a central board of menu items and kitchen locations (service discovery) rather than knowing each other's phone numbers. The manager also handles surges of customers by opening more stations (scaling) and closing them when quiet. This manager works all night, without sleep, and can handle a thousand dishes across fifty kitchens. But the manager also brings overhead—extra staff, extra paperwork, and a new set of rules. That is the cost of orchestration: for one kitchen, a manager is overkill; for fifty, it is essential. Where the analogy breaks down: the orchestrator is not a person, it is software that follows strict rules—it cannot improvise, and it does everything by reading configuration files. Also, the manager does not cook; it only coordinates. If a dish is poisoned (a bad image), the orchestrator will happily serve it again and again unless you tell it not to.


## Definition

Container orchestration is the automated management of containerised applications across one or many hosts, handling scheduling, health monitoring, restart policies, and service discovery so that services stay running with minimal human intervention.


## How it connects

You already know Docker Compose — that is the single-host orchestration tool. It declares services, networks, and volumes and runs them with one command. This topic builds on that: it takes the same ideas and scales them to multiple hosts and to automatic recovery. Your library notes on Backup and Hardware Failure are directly relevant: orchestration is what makes a service survive a hardware failure — if a host dies, the orchestrator moves the container to another host, provided you planned your backup reality. Your notes on Reverse Proxy and TLS also connect: service discovery often happens behind the reverse proxy, which uses the orchestrator's DNS names to route traffic. And your notes on Tailscale Mesh Networking: you can run an orchestrator over a mesh network, so you do not need to expose ports to the internet. This is also the foundation for later topics like Kubernetes, Docker Swarm, and autoscaling.


## The mathematics

Container orchestration does not rely on much mathematics — the core is logic and state machines, not equations. However, there is a small amount of probability and statistics underneath health checks and restart policies. Restart policies are essentially a way of handling uncertainty: a container might crash for a transient reason or a permanent one, and the orchestrator has to decide how many times to retry before giving up. Health checks produce a series of yes/no answers over time, and the orchestrator counts them to decide if a service is up or down — that is basic counting, which is arithmetic. The concept of 'backoff' in restart policies involves a geometric sequence: if a container keeps crashing, the orchestrator waits longer between attempts (e.g., 1 second, then 2, 4, 8, up to a cap). That is a sequence, and you can calculate the wait times by multiplying by 2 each time. So, the mathematics here is arithmetic plus a bit of sequences — nothing beyond what you already use daily. The branch of mathematics is discrete mathematics, which deals with counting, sequences, and logical decisions, rather than continuous calculus.

Let n be the number of restarts. The wait time before restart number n (starting from n=1) is: wait(n) = min(2^(n-1), max_wait) seconds. For this example, max_wait = 64, so the first waits are 1, 2, 4, 8, 16, 32, 64, and then it stays at 64.


## A worked example

```
Suppose an orchestrator detects that a web server container has crashed. Its restart policy is 'on-failure:5' — meaning it will restart the container up to 5 times before giving up. It uses exponential backoff: first it waits 1 second, then doubles the wait each time, up to a maximum of 64 seconds.

Step 1: Crash 1: wait 1 second, restart   # First restart attempt happens after 1 second.
Step 2: Crash 2: wait 1 × 2 = 2 seconds, restart   # Wait doubles.
Step 3: Crash 3: wait 2 × 2 = 4 seconds, restart   # Wait doubles again.
Step 4: Crash 4: wait 4 × 2 = 8 seconds, restart   # Still within the max of 64 seconds.
Step 5: Crash 5: wait 8 × 2 = 16 seconds, restart   # This is the fifth restart, which is the last allowed.
Step 6: It crashes again. The orchestrator gives up and stops restarting.   # The restart limit is reached, so the container stays down and alerts an operator.

Result: The container was restarted 5 times with increasing waits (1, 2, 4, 8, 16 seconds), then declared dead. Total time spent waiting: 1+2+4+8+16 = 31 seconds, plus the time to start each time.
```


## From scratch, in Python

```python
python
# from_scratch.py
# A tiny simulator of a restart policy with exponential backoff.

# We'll simulate a container that crashes after a few seconds of running.
# We use a function to represent the wait time before the next restart.
# (This is plain Python, no libraries.)

import time

def restart_policy(max_restarts, max_wait):
    """
    Simulate a container that keeps crashing.
    Returns the list of wait times (in seconds) between restarts.
    """
    restarts = 0
    wait_times = []
    while restarts < max_restarts:
        # Simulate the container running and then crashing after 1 second.
        print("Container is running...")
        time.sleep(1)  # pretend it runs for 1 second
        print("Container crashed!")

        # Compute the wait time as 2^(restarts), but cap it at max_wait.
        # restarts starts at 0, so first wait is 1, then 2, 4, 8, ...
        wait = 2 ** restarts
        if wait > max_wait:
            wait = max_wait

        print(f"Waiting {wait} seconds before restart...")
        wait_times.append(wait)
        time.sleep(wait)  # actually wait (but you can ignore this for a quick demo)
        restarts += 1

    print("Giving up: container is unhealthy.")
    return wait_times

# Run the simulator for max 5 restarts and a cap of 16 seconds.
waits = restart_policy(max_restarts=5, max_wait=16)
print("Wait times were:", waits)

```


## The same thing, with a framework

In real orchestration, you don't write this logic yourself – Docker and Kubernetes have restart policies built in. For example, in a Docker Compose file (which is a YAML file), you would add:

```yaml
services:
  web:
    image: nginx
    restart: on-failure:5
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost"]
      interval: 30s
      timeout: 10s
      retries: 3
```

This tells Docker to restart the container up to five times if it fails, and Docker itself uses exponential backoff (though it starts at 1 second and doubles, capped at 1 minute). The healthcheck tests if the service is actually responding; if it fails three times in a row, the container is marked unhealthy.


## What the code does

The Python code starts by defining a function called `restart_policy` — a function is like a recipe: it takes inputs (max_restarts and max_wait), does some steps, and returns a result. Inside, we use a `while` loop to repeat the steps as long as `restarts < max_restarts`. The loop simulates the container running (`print` a message, `time.sleep(1)` to pretend it runs for one second) and then crashing. Next, it calculates the wait time: `2 ** restarts` — that is two raised to the power of the number of restarts already done. So first time (restarts=0) it waits 1 second, then 2, 4, 8, etc. If the wait exceeds `max_wait`, we cap it. We then print and wait, and increment `restarts`. When the loop ends (after 5 restarts), we print a giving up message. The output is the list of waits. The framework version shows the Docker Compose YAML, which is the actual way you would configure this in practice — you write the policy, not the loop.


## Common misconceptions

**Orchestration is only for large-scale cloud applications** — I run a few containers on one VPS, so orchestration is overkill for me.

Even a single host benefits from restart policies and health checks — your services will restart automatically instead of staying down. In fact: Orchestration starts with simple restart policies and health checks that every self-hoster should use.

**Restart policies are the same as health checks** — If I set a restart policy, the orchestrator knows when a service is down.

Restart policy only triggers when the container exits, but a container can be alive and not responding (e.g., out of memory, hung). Health checks test ability to do work, not just process existence. In fact: Health checks are proactive probes (HTTP request, TCP check) that prevent sending traffic to a broken service.

**Service discovery is a DNS feature only** — Service discovery is just DNS, nothing more.

DNS resolves names to IPs but doesn't handle port changes, health-based filtering, or dynamic updates when containers start and stop. Orchestration also manages load balancing across replicas. In fact: Service discovery is DNS plus health-based routing and load balancing — a service name resolves to a set of healthy instance IPs.

**More replicas always means better performance** — Scaling up to more replicas always speeds things up.

If the bottleneck is a database shared by all replicas, adding web replicas just makes the database slower. Each replica also consumes CPU and memory that might be needed elsewhere. In fact: Scaling works only if the service is stateless and the underlying resource (DB, model) can handle the load.

**Orchestration replaces backups** — If my orchestrator moves my containers to a new host when one dies, my data is safe.

Orchestration recreates containers, not state. Volumes must be on shared storage, or data is lost. Your notes on Backup and Hardware Failure apply — a backup is only real once a restore has been tested. In fact: Orchestration provides durability of compute, but data durability requires replicated storage and proper backups.


## In short

Container orchestration is the automation that keeps your services running: it schedules containers on hosts, restarts them when they crash (with exponential backoff), health-checks that they are actually working, and lets services find each other by name. Start with restart policies and health checks in Docker Compose; later you can scale up to Kubernetes. The heavy lifting is operational — the mathematics is just counting and doubling.


## Where this is used

Container orchestration is central to deploying and running AI systems — not the mathematics of the models, but the operations. For large language models (LLMs): when you self-host a model like LLaMA or Mistral on a cluster of GPUs, you use an orchestrator (most often Kubernetes) to schedule the model serving pods across the GPU nodes. The orchestrator handles GPU resource requests, ensures the model server stays up (health checks on the inference endpoint), and scales the number of replicas up and down as request load changes. For a single large model that doesn't fit on one GPU, you might use the orchestrator to manage the distributed sharding of the model across multiple GPUs. For small models running on devices (edge AI, TinyML): orchestration is lighter, but still present — a home assistant that runs wake-word detection on a Raspberry Pi uses a simple orchestration layer to restart the service if it crashes and to update the model without downtime. The orchestrator may be as simple as a systemd unit file (a Linux service manager that does restart on failure) — that is a minimal orchestrator. For retrieval-augmented generation (RAG): a RAG pipeline typically has multiple components: an embedding model, a vector database, and an LLM. Each runs as a separate container. Orchestration ensures they are all up and discoverable. Service discovery is crucial here: the LLM container needs to find the vector database by its service name (e.g., `vector-db:5432`), not a hardcoded IP, so you can move the DB to another host without changing the LLM's config. For agentic systems that plan and call tools: an agent loop is often a long-running service. If it crashes mid-task, the orchestrator restarts it, but the state may be lost unless you implement checkpointing. The orchestrator also manages the tool-calling backend services (e.g., a code interpreter container) — it can spin up a fresh container for each tool call and tear it down afterwards, which is a common pattern. For everyday applied ML (recommendations, forecasting, fraud detection): model serving services are wrapped in containers and orchestrated to maintain uptime and auto-scale based on request rate. For example, a recommendation service might run 10 replicas during the day and scale down to 2 at night. In practice, a developer writes a Kubernetes Deployment YAML with a resource request (CPU/memory), a health check hitting the model's /health endpoint, and a HorizontalPodAutoscaler that scales based on CPU usage. They might use `kubectl` to apply it. What breaks when done badly: if health checks are too lenient, the orchestrator thinks a broken service is fine and traffic goes to a dead endpoint; if restart policies are too aggressive, the orchestrator thrashes (restarts containers repeatedly), making the system worse. The number one failure is not setting a reasonable `max_restarts` and backoff, which leads to a crash loop that burns CPU and memory.


---

Read this on the web: https://basavapurushottam.com/learn-ai/container-orchestration-basics
