# Backup and Hardware Failure

Planning for disks and machines failing — backup strategy, offsite copies and tested restoration.

**Rests on:** Container Orchestration Basics

*MLOps and Platform · Self-Hosting*


## Why learn this

This concept is the foundation of self-hosting responsibility. Without it, you lose everything when a drive fails — and drives always fail. Mastering backup strategy, offsite copies, and tested restoration unlocks the rest of your self-hosting journey: Docker Compose, container orchestration, reverse proxies, and even local model hosting. Every one of those assumes your data survives hardware failure. Once you can restore from nothing, you can experiment fearlessly — upgrading, breaking, and re-building your stack without losing what matters.


## The intuition

You own your hardware. You also own its failures. A hard drive is a mechanical or electronic device with a finite life — it will stop working eventually. When it does, everything stored on it — your files, your Docker volumes, your database — is gone unless you planned for that moment. Planning for hardware failure means making copies of your data somewhere else, and checking that those copies actually work. The word 'backup' is often used loosely. A copy on the same disk is not a backup — if the disk dies, both copies die. A copy on an external drive in the same room is better, but a fire, flood, or theft takes both. The real protection is an offsite copy — somewhere physically separate from your machine. But even an offsite copy is worthless if you have never tried to restore from it. The first time you need to restore is the worst time to discover your backup is corrupt, incomplete, or unreadable. So the practice is: take regular snapshots, send them somewhere else, and test a restore at least once. That test is what turns a hope into a guarantee.


## An analogy

Imagine you keep your only handwritten manuscript in a single notebook on your desk. One night, the desk is destroyed in a fire. The notebook is ash. That is what happens when you have no backup. Now imagine you photocopy the manuscript every day and keep the photocopy in a drawer across the room. The fire takes the desk but not the drawer — but if the drawer is in the same building, a flood or a burglary could take both. So you also mail a weekly copy to a friend in another city. Now you have three tiers: the working notebook, a local copy, and an offsite copy. But here is the trap: you never opened the mail copy. When the fire happens and you go to your friend's house, you find the photocopies are blurry, pages are missing, and the mail service lost half the package. You thought you had a backup; you had a stack of paper. The analogy holds for your digital life. Your working files are the manuscript. A backup is the photocopy. An offsite copy is the mailed package. And a tested restore is you, opening the package, checking every page is there and readable, before the fire. The analogy breaks down in one way: digital copies are cheap, so you can afford to back up far more often than daily, and you can automate it. But the human habit of neglecting the test remains exactly the same.


## Definition

Backup and hardware failure is the practice of planning for the inevitable loss of a storage device by maintaining redundant, offsite copies of your data and regularly verifying that those copies can be restored successfully.


## How it connects

This concept sits at the foundation of your self-hosting journey. You have already chosen to take control of your services and data — 'total control means total responsibility' — and this is where that responsibility bites hardest. Security and hardware failure become yours, as your notes say. Your other self-hosting topics — Docker Compose, container orchestration, reverse proxies — all rely on persistent data stored in volumes. If the disk hosting those volumes fails, the entire stack dies. Backup is what lets you recover. When you learn about container orchestration, you will see how restart policies and health checks make services self-healing, but those recover from software crashes, not hardware failure. Only backups recover from a dead drive.


## The mathematics

This concept does not rest on a branch of mathematics like algebra or calculus. It rests on arithmetic and simple ratios. The only 'math' you need is counting and comparing sizes. You compare the size of your data to the capacity of your backup location, and you compare the time between backups to how much data you are willing to lose. For example, if you back up every night at midnight, a failure at 11 PM means you lose up to 23 hours of changes. That is your 'recovery point objective' — the maximum acceptable data loss. You also count how many copies you keep, and where they are. The arithmetic of storage is simple: a terabyte (TB) is 1000 gigabytes (GB), a GB is 1000 megabytes (MB). When you plan a backup, you add up your data sizes and check the backup disk has room.

Storage needed for N full backups = Data size × N. Recovery Point Objective (RPO) = time between backups. Recovery Time Objective (RTO) = time from failure to restored service.


## A worked example

```
Your data: 500 GB. Your external drive: 1 TB (1000 GB). Your offsite cloud account: 2 TB (2000 GB). You want to keep 7 daily backups.

Step 1: 500 GB (data) × 7 (daily copies) = 3500 GB   # If you keep 7 full copies of your data, you need 3.5 TB of space, which exceeds your external drive's 1 TB. You cannot keep 7 full copies there.
Step 2: 500 GB (data) × 1 (external drive copy) = 500 GB   # One full copy of your data fits on the external drive (500 GB < 1000 GB).
Step 3: 500 GB (data) × 7 (daily offsite copies) = 3500 GB   # The cloud account has 2 TB, so it also cannot hold 7 full copies. You need a different strategy, like incremental backups that only store changes.

Result: Your current plan of 7 full copies exceeds both backup locations. You must either reduce the number of copies, compress the data, use incremental backups, or buy more storage. The arithmetic shows you the constraint before you hit it.
```


## From scratch, in Python

```python
import os
import shutil
import datetime

# A backup is just copying files. Here's a simple script that does it.

def backup_directory(source, destination):
    """Copy everything in source into a timestamped folder in destination."""
    timestamp = datetime.date.today().isoformat()  # e.g. '2025-03-28'
    backup_folder = os.path.join(destination, timestamp)
    if not os.path.exists(backup_folder):
        os.makedirs(backup_folder)
    for item in os.listdir(source):
        full_path = os.path.join(source, item)
        if os.path.isfile(full_path):
            shutil.copy2(full_path, backup_folder)
        elif os.path.isdir(full_path):
            shutil.copytree(full_path, os.path.join(backup_folder, item))
    print(f"Backed up to {backup_folder}")

# Example usage:
# backup_directory('/home/me/data', '/mnt/external/backups')

```


## What the code does

This script uses Python's `os`, `shutil`, and `datetime` libraries — these are built-in, no installation needed. The function `backup_directory` takes two arguments: `source` (where your data lives) and `destination` (where backups go). It creates a folder named with today's date inside the destination, so each backup is kept separately. Then it loops through every file and folder in the source. For files, `shutil.copy2` copies the file preserving metadata. For directories, `shutil.copytree` copies the whole folder and its contents. Finally, it prints a confirmation. This is the core of any backup: copying files to a different location. Real backup tools (like `rsync`) do the same but handle large data more efficiently, sending only changed parts. The script demonstrates the principle: a backup is a copy, and a timestamped copy helps you keep history.


## Common misconceptions

**My RAID array is a backup** — If I have two disks in RAID 1, my data is safe because the second disk mirrors the first.

RAID protects against a single disk failure, but not against controller failure, power surge, theft, or a logical error that deletes data on both disks simultaneously. RAID is not offsite. In fact: RAID is high availability, not backup. Use RAID for uptime, but still maintain regular offsite backups.

**Copying to an external drive once is enough** — I copied my files to an external drive last month, so I'm fine.

The drive might fail, be lost, or be corrupted. The copy is a snapshot in time — any changes since then are unprotected. And you have not verified the copy works. In fact: Back up regularly, keep multiple copies, and test restores. The 3-2-1 rule: three copies, on two different media, one offsite.

**Cloud storage is forever** — My files are in Google Drive, so they're safe forever.

Cloud providers can delete your account, suffer outages, or you might forget to pay the bill. Your data is not yours if you depend on a third party. In fact: Treat cloud storage as one of your backup targets, not the only one. Download and verify regularly, or use a tool like restic to encrypt and sync.

**Backup is a one-time setup** — I set up a backup script, so I'm done.

Data changes, new services get added, volumes grow. Backup scripts can silently fail due to permissions, disk full, or syntax changes. A backup that never runs is a myth. In fact: Schedule backups, monitor that they run, and periodically test restores. Automate and check the logs.

**Restoring is just copying files back** — I can just unzip my backup to get everything back.

Databases are often inconsistent if you copy the raw files without using the database's own dump command. Configuration files may need specific permissions. A backup without a tested restore procedure is a trap. In fact: Use the right backup method per service (e.g., pg_dump for PostgreSQL), document the restore steps, and rehearse them.

**Small data doesn't need backup** — I only have a few hundred megabytes of essential files, so I can just keep them on a USB stick.

Small data is the most likely to be lost because it is easy to ignore. A USB stick is fragile and easy to misplace. A single copy is not enough. In fact: Even small data deserves the same discipline: multiple copies, offsite, tested. It costs nothing to do it right.


## In short

Backup is not a tool, it is a habit: copy your data regularly, send copies offsite, and prove you can restore them. Hardware failure is certain; survival is not. The 3-2-1 rule — three copies, on two different media, one offsite — and a tested restore are the minimum for a responsible self-hoster.


## Where this is used

In self-hosting, you are the system administrator. Your services — whether a blog, a Nextcloud instance, or a local language model — store data in Docker volumes. Those volumes live on a disk that can fail. Your backup strategy protects that data. In practice, developers use tools like `rsync` for Linux, `restic` for encrypted offsite backups, or `borgbackup` for deduplication. They also use Docker's named volumes and bind mounts to make data persistent, then back up the volume contents. For example, a docker-compose file might declare a volume for a database, and a cron job runs a backup script every night that uses `pg_dump` for PostgreSQL or `mongodump` for MongoDB before copying the dump to an external drive. Then the script uploads the file to an offsite cloud storage (like S3) using `aws s3 cp`. The critical step is testing a restore: a developer will spin up a fresh container with the same volume mount, run the restore command, and verify the data is there. If the restore fails, the backup is worthless. In LLM deployments, model weights are large files (several GB) — backing them up is just copying, but if you lose them, you cannot regenerate them easily. For RAG systems, an embedding database (like a vector store) is the data — if lost, the entire knowledge base must be re-embedded, which takes time and API costs. Agentic systems log every tool call and action — losing those logs means losing the ability to audit or replay what happened. Therefore, backup is not an afterthought; it is part of the deployment. Developers write scripts to snapshot volumes, use tools like 'duplicati' or 'restic', test restores quarterly, and document the recovery procedure. A good backup strategy is boring: automate it, test it, and don't think about it until you need it.


---

Read this on the web: https://basavapurushottam.com/learn-ai/backup-and-hardware-failure
