By Matija, the developer behind Build with Matija. In the first week of November 2026 I replaced a set of per-project backup scripts on my main server with one backup manager that covers 12 PostgreSQL databases and 8 Garage object stores across my own products and client sites. This article is written from that work: the pull requests, the recovery and setup docs, the test suites and the fixes that the first real runs forced. Versions: PostgreSQL (the pre-rollout rehearsal mentioned below used 18.6), Garage with its SQLite metadata engine, rclone with the crypt and Google Drive backends, Infisical self-hosted, Docker Engine on Linux. Last updated 2026-11-09.
A backup on a self-hosted server counts only when you can show three things: the dump is complete, the off-site copy matches it byte for byte, and a restore of it produces a working database. My setup checks all three on every run. A dump is published only after pg_restore -l accepts it, the encrypted upload is verified with rclone cryptcheck, a manifest.json is written last so a half-finished run never looks like a backup, and once a month one database is restored into a throwaway container and its row counts are compared with the counts taken at backup time. The same pipeline covers Garage buckets, including their metadata. Below is the design, the bugs that only showed up on real data, and the recovery order you need when the server that holds your secrets is the server you lost.
Why the old backups were not enough
Before November every stack on the server looked after itself. The Postgres starter I use for new projects shipped a backup.sh and a cron line that up.sh installed, a few products had their own gpg-encrypted jobs, and the Garage instances had no off-site path at all. Each piece worked on the day it was written. Together they had no single answer to "what is backed up, when did it last succeed, and could I restore it".
The first pull request in the new work found a bug that had been there all along. The starter's backup.sh redirected pg_dump straight into the final file. When a dump failed halfway, the truncated file stayed on disk with a normal name and a normal timestamp, and it looked exactly like a backup. Retention would then happily keep it and delete older, good ones.
If you followed my guide to self-hosting Next.js and Payload on a VPS, its pre-migration step uses the same pg_dump > file pattern, so the fix below applies there too.
The split: dump tools stay with the stack, one manager owns everything after
I kept the knowledge of how to dump a particular thing next to that thing, and moved everything after the dump into one place.
The backup manager is a small Python CLI with no dependencies outside the standard library. It runs on each host and pushes straight to the destination, so there is no central backup server to keep alive. The policy file is the one authority on what runs when:
toml
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
Each source says where its credentials come from and which group it belongs to. max_age_hours = 30 is what status --fail-on-stale checks against, so a nightly job that silently stopped shows up the next day. keep_min_verified = 3 means retention can never delete the last three verified runs, whatever the age limit says. When the manager registers a stack, the starter's setup-backup.sh removes that stack's old cron line and refuses to reinstall it, which stops two schedulers from backing up the same database. That handoff only happens after the stack has a verified off-site run.
The inventory commands mattered more than I expected. My Garage instances live in four different folder layouts, so the bootstrap repository could not find them by reading its own local/ directory. garage-discover asks Docker instead, and whcp-backup inventory prints every database and bucket on the host against the policy, with UNREGISTERED next to anything nobody has decided about yet.
Make a failed dump impossible to mistake for a backup
The fix for the truncated-dump bug is a write-then-promote pattern. The dump goes to <output>.partial, and the file is renamed to its real name only after pg_dump exited cleanly and pg_restore -l accepted the archive. The check runs inside the database container, so the client tools match the server version. A failed run leaves nothing behind.
Here is the idea in its smallest form, for adapting into your own script:
The real script adds a per-target lock and an optional --globals export, and gives every failure its own exit code so the manager can report what went wrong without parsing text:
The globals file is easy to forget. A pg_dump archive holds one database and none of the cluster's roles or their password hashes. If you restore into a fresh container without them, ownership and grants fail. The manager stores *.globals.sql with every run, applies it on scratch restores, and uses --no-owner --no-privileges when restoring into an existing container.
Its test suite runs 27 checks against a throwaway container and a fake crontab, including truncated dumps, archives that fail validation, two runs competing for the lock and the cron ownership handoff.
A run publishes its manifest last
Every run of every source follows the same order:
Take the per-source lock. A second run of the same source is recorded as skipped.
Check free disk space.
Export through the source's own repository. A failed export publishes nothing.
Upload through an rclone crypt remote, then verify the upload with rclone cryptcheck.
Upload manifest.json last.
Step 5 is the rule the rest depends on. A run folder at the destination without a manifest is treated as not a backup by list, restore and retention alike. An upload that died at 90 percent therefore cannot be restored by accident, and it cannot push a good run out of the retention window.
Scheduling is a single cron line that runs whcp-backup tick every five minutes. The tick evaluates each source's schedule in the policy's timezone, runs due jobs one at a time, catches up on anything missed while the host was down, and retries a failure an hour later, twice by default. The crypt remote encrypts file and folder names as well as contents, so the Drive UI shows only ciphertext, and list is how you see what is there.
Restore drills are part of the schedule
Once a month the tick restores one Postgres source into a scratch container, rotating through them, so every database gets drilled in turn. You can also run it by hand with whcp-backup verify <source> --restore.
The scratch container is chosen from the run's manifest: the same image when it can be pulled by name, otherwise the same major version with the extensions the database needs. A database that uses PostGIS needs a PostGIS image, and pgvector needs a pgvector image, so the manifest records them. After the restore, the manager compares row counts with those taken at backup time. A missing table, or a table that came back empty, fails the drill. Small differences on busy tables are reported as row_drift without failing.
I had already used the same habit before risky changes. Before turning on single sign-on in Docket in production, its run report records a backup taken, pg_restore -l listing 41 table-data entries, and a restore into a throwaway PostgreSQL 18.6 where the new migration was rehearsed first. My identity server goes further and has scripted drills for restoring the database to a fresh PostgreSQL with the same signing keys, for a wrong secret, for an emergency key rotation and for a database outage, all against throwaway Docker resources.
Each Garage run uploads a native garage meta snapshot with the config, node files, layout, bucket list and key list, and mirrors the bucket contents into a current/ folder, with a history/<run>/ folder per run alongside it. The backup reads through an additive read-only key granted on every bucket, and the integration tests prove that key cannot write. Which recovery you run depends on what you lost:
Two guards protect the mirror itself. An empty-source guard makes sure a source that suddenly reports no objects, or is broken, can never wipe the copy, and the integration tests delete a bucket's contents on purpose to check it. And an empty bucket is a valid state with no folder at the destination, so restoring it must succeed with nothing copied:
python
# File: whcp_backup/adapters/garage.py (excerpt)
current = destination.path(*base, "current", args.bucket)
# An empty bucket has no folder at the destination, and that is a valid state.if rclone.lsjson(current, "--max-depth", "1", missing_ok=True):
rclone.copy(current, into)
The Garage integration suite runs 40 checks on throwaway containers, covering both recovery scenarios, the read-only key and the empty-source guard.
A database and its bucket are not transactionally consistent with each other. A group in the policy runs an app's database and then its bucket back to back, which narrows the gap. After restoring both, a row can still point at an object written after the bucket mirror, or the other way round, and your app should tolerate that.
Four things that broke on the first real runs
All sources started disabled, and the tests passed against throwaway containers. Turning on the real 12 databases and 8 buckets on 5 November found four problems the tests had missed.
A container that vanished mid-scan failed unrelated backups. The inventory lists containers with docker ps and then inspects each one. A short-lived container removed between the two calls made docker inspect fail, which crashed the whole inventory, and that failed real backups of two Garage instances that had nothing to do with it. The fix in both dump repositories was to skip a container that is gone by the time it is inspected.
Google Drive reports a missing folder as exit code 3 with an empty error. The first mirror of every Garage bucket checks the destination folder, which does not exist yet. At the log level I use, rclone returns exit 3 and prints nothing to stderr, so a check that looked for "not found" in the error text treated it as a failure. The manager now treats exit 3 as "missing" wherever a missing folder is acceptable.
Scratch restores left their volumes behind. Removing a test container with docker rm -f leaves its anonymous data volume on disk, so a monthly drill would leak a database-sized volume every time it ran, which is a slow way to fill a disk. Every cleanup now uses docker rm -f -v.
An expired Drive token looked like a generic upload error. If the Google OAuth consent screen is left in Testing mode, refresh tokens expire after seven days and every backup starts failing. The manager now recognises invalid_grant and says what to do:
python
# File: whcp_backup/rclone.py (excerpt)if"invalid_grant"in result.stderr:
raise StepError("auth", "Google rejected the Drive token. Run bin/authorize-drive to sign in again, and check the OAuth consent screen is In production.")
Setting the consent screen to In production needs no Google verification here, because the remote uses the drive.file scope, which only lets the token see files it created.
Recover the secrets first
Every secret on this server comes from a self-hosted Infisical, which I set up as described in Self-Host Infisical: Ultimate VPS Guide for Secrets. The backup manager reads its rclone remotes, crypt passwords and Drive token from Infisical as RCLONE_CONFIG_* variables, and since 6 November new Garage instances store their S3 credentials only in Infisical as well. The only credential on disk is the machine identity file, mode 600.
That creates the trap at the centre of disaster recovery on one box: Infisical runs on the same disk it is protecting. If the disk goes, a key that exists only in Infisical goes with it, and every encrypted backup becomes unreadable forever. The setup guide therefore makes one step mandatory before any real source is enabled: save the recovery kit outside the host. A small script prints it, and it refuses to write anywhere except a real terminal, so the keys cannot end up in a log file or a pipe:
bash
# File: bin/show-recovery-kit (excerpt)if [[ ! -t 1 ]]; thenecho"Run this in a terminal: it refuses to write keys anywhere else." >&2
exit 1
fi
The kit holds the two crypt passwords, the Google OAuth client and the passphrases of the older gpg jobs, including Infisical's own encrypted backup. With it, recovery after total disk loss runs in a fixed order:
Install Docker on a new machine, clone the backup manager and install rclone.
Reach the backups without Infisical by authorizing Drive again and exporting the remote for that shell only. Use the same OAuth client ID: a drive.file token can only see files created by its own client, so a new client sees an empty Drive. If the client is gone, authorize with the full drive scope and switch back afterwards.
Step 2 is the one I would have got wrong under pressure. Nothing in the Drive UI tells you that your new token cannot see the backups because a different OAuth client made them.
What is still missing
There is no push alerting yet. A failure is visible in whcp-backup status, tick.log and status.json, and nobody is told. status --fail-on-stale is built for a monitor to call, and wiring it to one is the next step.
The Garage metadata restore only supports SQLite metadata and refuses anything else, and one older instance has no db_engine in its config, so its snapshot format is not confirmed yet.
FAQ
Is pg_dump alone a backup?
It is the first of three steps. You also need proof the archive is complete (pg_restore -l on the file), proof the off-site copy matches it (a checksum or rclone cryptcheck), and a periodic restore into a fresh database with a row-count comparison. Without the last step you are trusting a file you have never opened.
Why write the manifest last?
So a run that dies halfway is never mistaken for a good one. If listing, restore and retention all ignore runs without a manifest, a partial upload cannot be restored by accident and cannot cause retention to delete a good run.
Do I need to back up Postgres roles separately?
Yes. A database dump holds no cluster roles or password hashes. Export them with pg_dumpall --globals-only next to each dump and apply them before restoring into a fresh cluster.
How do I back up Garage?
Back up two things: the metadata, with garage meta snapshot plus the config and layout, and the objects, by mirroring each bucket through a read-only key. Metadata alone recovers from corruption. A full disk loss needs the objects too, restored into a fresh instance with new credentials.
What happens if my secrets manager is on the same server?
Keep the decryption keys and the backup destination's credentials outside it, in a password manager, and restore the secrets manager first. Otherwise a disk failure takes the only copy of the keys with it.
Conclusion
A self-hosted server that runs many products needs backups it can prove: a dump that is validated before it gets its real name, an encrypted copy that is checked after upload, a manifest that marks a run as complete, and a restore drill that runs on a schedule and compares row counts. Splitting the work so each stack's own tooling makes the dump while one manager owns scheduling, verification and retention gave me a single place to answer what is backed up and whether it restores. The first real runs still found four bugs no test had, and the recovery order starts with the secrets because that is the step that otherwise ends the recovery before it begins.