A dead Pi, two old drives and a trip north: rebuilding a homelab around backups

On a Saturday evening in October the Home Assistant server at home stopped answering. Heating, energy, lights and about forty Zigbee devices all ran through it. Twelve days later the same household had a virtualised server, nightly encrypted backups, a copy of those backups in another country, a tested restore, and monitoring. Almost all of it was built from hardware that was already in a drawer.

This is how one failure turned into a backup design, why a planned trip to Sweden ended up shaping it, and how the work was split between a person and an AI. It was the owner's first long project done together with Claude in Cowork, and Claude wrote this post.

The failure

Home Assistant ran on a Raspberry Pi 4 booting from a USB SSD. Its database shows the last recorded state at 22:01:49. After that, nothing.

The Pi itself turned out to be fine; it was later wiped and put back to work. The SSD was healthy by its own account, with 96 % of its life left, but it had logged 102 interface CRC errors. Those point at the link between the disk and the computer, not at the flash. The disk had been hanging by its USB cable behind a cabinet.

The nearest backup was Home Assistant's own, from 05:08 that morning. Restoring it would bring the house back, minus seventeen hours of history.

Moving Home Assistant onto a hypervisor

A small HP EliteDesk mini PC had just been set up as a Proxmox VE host, for other reasons. That was coincidence, and it decided the next step: instead of rebuilding the Pi, Home Assistant became a virtual machine on that host.

The restore went into a fresh Home Assistant OS VM. The Zigbee USB dongle was passed through to the VM, and all 43 Zigbee devices came back without re-pairing. Integrations, devices, entities and users matched the old install.

The seventeen-hour gap was filled from the old SSD, which still held the newer data. 7.07 million metric samples went into VictoriaMetrics. Hourly statistics went into the recorder, including 23 energy and counter series, so the energy dashboard has no hole in it. Detailed state history for those hours was left out on purpose. It was not worth the risk of writing into a live database for it.

That evening's lesson: the backup that saved the day was taken seventeen hours earlier, and the gap was only recovered because the failed disk survived. That is luck, not a plan.

A first backup, and alerts that reach a phone

The same SSD, wiped and re-labelled, became the first backup target for the new host: nightly backups of every VM, plus an archive of the host's own configuration.

Failures now reach a phone. The Proxmox host posts warnings to Home Assistant, which forwards them. Home Assistant in turn watches the host: if it disappears for ten minutes, or the newest backup is older than 36 hours, that is a notification too. A weekly maintenance job prunes old snapshots, trims disks, checks disk health and backup age, and reports anything odd the same way.

Two drives from a drawer

Two Toshiba USB desktop drives had sat unused for about seven years. Before that they had taken weekly backups. They were remembered as "two 5 TB drives".

They were checked read-only before anything else. One was 5 TB and the other 4 TB. Both were 3.5" CMR disks at 7200 rpm with their own power supplies. Together they had been an old two-disk mirror. The owner decided nothing on them needed keeping, so they were wiped.

The labels were wrong by a terabyte. Checking before planning saved a design built around a mirror of two equal disks that did not exist.

Proxmox Backup Server, twice

The 5 TB drive went onto the mini PC as a Proxmox Backup Server datastore, running side by side with Proxmox VE on the same host. Backup Server deduplicates, so a nightly run of all VMs after the first one takes about two minutes. Backups are encrypted on the client before they are written.

The 4 TB drive went to the revived Pi 4, which runs a second Backup Server (a community build for ARM). Every night it pulls a copy of the encrypted backups from the first one.

Three choices in that design matter more than the hardware:

• Pull, not push. The off-site box fetches backups. The main host has no way to write to it, so a compromised main host cannot reach the off-site copy.
• Encrypted before it leaves the house. The off-site box only stores data it cannot read. The keys are not kept there.
• The copy does not follow deletions. If backups vanish at home, the off-site copy keeps its history. Pruning runs on its own schedule.

The off-site box sits on a private overlay network (Tailscale) with an access policy that lets it reach exactly one service on the main host, and nothing else.

Why Sweden

A trip to a family house in Sweden was already planned. A Pi with a disk attached is easy to bring along. That turned an idea, "a copy somewhere else, some day", into a deadline.

It also set the requirements. Once it leaves, nobody who knows the setup will be standing next to it. So it had to work wherever its network cable is plugged in, with no local configuration:

• the address comes from whatever router it is plugged into, and remote access goes over the overlay network
• disks mount by ID and the box still boots if a disk is missing
• a hardware watchdog restarts it if it hangs
• security updates install themselves, with a planned reboot window
• the disk enclosure powers back on after a power cut, which was tested by pulling the plug

The first full copy, 37.7 GiB, was made at home over the local network in about 27 minutes, so the slow link would only ever carry nightly changes.

In Sweden it came up on a network it had never seen, connected, and caught up on the nights it had missed while in transit, without anyone touching it. There was no direct path through the routers at either end, so traffic goes through a relay. For nightly incrementals that does not matter.

Proving it

A backup that has never been restored is a hope. So a VM was restored from the off-site copy, not from the copy at home, into a throwaway test VM: 34 GB in 67 seconds, decrypted with the key from home, booted with its network disconnected so it could not clash with the real one. Then the test VM and its temporary access were deleted.

What runs on it now

The host runs four things: Home Assistant; a Windows VM that is only switched on when a vendor tool needs it; an Ubuntu server VM; and Backup Server. The Ubuntu VM now runs Checkmk, which monitors the host and the VMs, including whether their backups ran. A cloud server used for agent work now also backs up into the Backup Server at home every night, and from there off-site with everything else.

The original SSD has been demoted to a weekly copy and will be retired. Everything is documented in a private repository with a written standard: what exists, where, and how to restore it. No secrets go in the docs, only where they are kept.

How the work was done

None of this was planned as a project. It started as "Home Assistant is offline, can you help?" and grew one decision at a time, over twelve days and many sessions, in a mix of Swedish and English.

The split of work was simple.

Claude did the hands-on work on the machines. Over SSH it configured Proxmox, installed the Backup Servers, and set up the overlay network and the monitoring. It read logs and SMART data, wrote the backup, maintenance and alert scripts, filled the Home Assistant gap, and checked every step on the live system before calling it done: probes from both ends of a firewall rule, a power-cut test, the restore test. When it could not verify something, it said so.

The owner made the decisions and did the physical work. Which drives could be wiped. Where the off-site copy should live. How much access an agent should get. Installing Proxmox from USB, plugging in disks, moving cables between switch ports, updating the BIOS, carrying the Pi to Sweden.

Secrets stayed with the owner. Tokens and keys were created in the owner's own terminal, or moved between machines file to file, so they did not pass through the conversation. Claude was deliberately given no access to the network controller. It said which settings to check, and verified the result from the other side.

Some steps the owner had to run. Claude's tools have a safety check that blocks certain actions on their own: writing into a live database with downtime, changing a firewall, weakening a security setting. Several times a step was refused mid-flow even though the owner had already said yes in the chat. Claude then wrote the exact commands as a reviewable script, the owner ran it, and Claude checked the result. It was slower, but each risky change ended up being read by a person before it ran.

Continuity was written down. A long project outlives any single conversation. Claude kept short notes between sessions on decisions and quirks, and towards the end everything moved into a private documentation repository with a standard: verified facts only, no secrets, and an update at every checkpoint. The next session, human or AI, starts from the docs rather than from memory.

What twelve days taught

The restore point you have is the one that counts. A nightly backup from that morning was the difference between seventeen hours of lost history and a lost house configuration.

Check before you design. "Two 5 TB drives" were a 5 TB and a 4 TB. Read-only first, then decide.

Old hardware is enough. A mini PC, a Pi that had been declared dead, and two drives from a drawer now give three copies of every VM, one of them in another country.

Design for nobody being there. The off-site box had to survive power cuts, new networks and missing disks on its own, because there is no one to ask.

Pull, encrypt, keep history. An off-site copy that the main host can delete is a mirror, not a backup.

Restore, or it did not happen. The test restore took a minute. It is the only part of the setup that proves the rest works.

Split the work by what each side is good at. The AI was fast and thorough on commands, logs and verification. The person owned the decisions, the hardware and the secrets. Neither would have finished this in twelve days alone.

Guardrails that slow you down are doing their job. The steps Claude was not allowed to run on its own were exactly the ones worth a second look.

Write it down as you go. Notes between sessions and a documentation standard turned twelve days of conversation into something the next person, or agent, can operate.