Backups

An archive with no shape.json restores normally: the source shape reads as unknown and the restore reconciles against the target regardless.

What a restore reconciles

Every restore path — the archive, the chain, and a From-Backup clone — ends in reconcile_site.sh, which makes the restored site agree with the machine it landed on. It reports each value it changed and refuses rather than papering over a mismatch it cannot fix.

The identity, in both places it lives:

Two files in a backup are the machine's, not the backup's, and the target's own copies survive every restore:

• config/Globalvars_site.php holds this machine's database password and its secret_box_key. Restoring the source's copy is what turns a clean-looking rebuild into SQLSTATE[08006] on every page — and it bites a same-shape rebuild exactly as hard as a cross-shape one.
• config/backup_site_key identifies one machine as a recipient of its own backups. Two machines sharing it means one machine's key opens the other's archives. backup_envelope.php mints a fresh one on first use, so absent is the correct state.

When the captured virtualhost differs from the generated one it is kept beside the live file as {site}.conf.from-backup and named in the output, so a hand-added redirect or alias survives on disk without being applied unattended.

The certificate is never waited for. The reconcile arms joinery-ssl-retry@{domain}.timer (arm_ssl_retry.sh) and disarms the old domain's. That timer checks DNS every five minutes, does nothing until the domain resolves to this server, then issues once and disables itself. The <IfFile> guard on the :443 block means the site serves HTTP until then rather than Apache refusing to start. Restore now, cut DNS later, certificate arrives on its own.

How chains work

The files archive uses GNU tar's --listed-incremental against a snapshot file at {working dir}/.{slug}.snar, where the working directory is the profile's own. tar records each directory's full contents, so restoring replays deletions as well as additions — a file removed last Tuesday is absent when you restore to Wednesday, rather than rising from the dead.

The archive is taken from the live tree, not from a staging copy. This is not a preference: an rsync copy gives every file a new ctime, so tar sees the whole site as changed and every "incremental" silently becomes a full. Measured, and pinned by a test.

Archiving a live tree means a file can change while tar is reading it. That is tolerated — GNU tar reports it with exit status 1, the run notes it and carries on, and the file's settled version ships with the next run. It also means the file set has no single point-in-time: the database dump is taken minutes after the file archive, so a deploy landing mid-backup can leave the two slightly skewed. For a web tree this is the normal trade; restore the newest run if it matters.

The database is dumped in full on every run. A dump is the small part, and a half-applied database is not something anyone wants to restore.

A run starts a new chain when there is nothing to extend, when the snapshot file is missing or empty, when the chain is older than the configured interval, when one full is carrying more than 30 incrementals, or when the chain's envelope no longer opens with the site key (the site key is disposable; a chain sealed to a lost one cannot be extended, only restored). Losing the snapshot — or the local manifest — is therefore safe: the next run costs one extra full, and never produces a broken backup.

A run that fails partway clears the snapshot for the same reason: the snapshot advances while tar runs, before the run is committed to the manifest and confirmed in the bucket, so carrying it past a failure would quietly leave the failed run's changes out of the chain. The next run starts a fresh chain instead — one extra full, never a silently broken backup. The failed run also deletes its own artifacts and puts the manifest back to its pre-run state: the abandoned run's archives can be gigabytes a small disk does not have to spare until chain retention removes the chain, and a local manifest describing a run the bucket never received must not survive to be uploaded by anything later.

Runs are serialized with a lock in the working directory; a run that finds another in progress reports itself skipped rather than racing it for the snapshot and the manifest.

Retention over chains is atomic: a chain is kept or deleted whole. Deleting the oldest runs of a chain would leave incrementals whose full is gone, which is not a smaller backup — it is no backup, and it would look like a restore point right up until someone needed it.

Restoring a chain

php maintenance_scripts/sysadmin_tools/backup_envelope.php open \
    --sidecar manifest.json --private ~/recovery.key --key-out /tmp/k
bash maintenance_scripts/sysadmin_tools/restore_chain.sh {project} \
    --artifacts {downloaded chain dir} --key-file /tmp/k [--seq N] [--domain d]

The plaintext data key exists only as a 0600 file for the length of the run and is destroyed before the run ends. It is never passed in argv, and on the fleet path it never enters a management job row.

config/backup_site_key is pinned to 640 www-data:www-data by fix_permissions.sh, and minted at the same mode. Backups run under more than one account — the web user takes the scheduled run, the deploy account runs one from a shell — and both are in the www-data group, so the key is group-readable rather than owner-only. A key that exists but cannot be read is an error, never treated as absent — minting over a live key would orphan the site recipient for every backup already sealed to the first one.

Opening a backup with no Joinery anywhere

The format is deliberately stock crypto, so a backup opens on any machine with openssl and PHP's sodium extension (or any libsodium binding) — no Joinery code, no server, no network. You need three things: the encrypted archive, its envelope (the {archive}.keys.json sidecar beside a standalone archive; for a chain, the envelope object inside the chain's manifest JSON), and the recovery private key from the password manager (base64, 32 bytes decoded).

Step 1 — unseal the data key from the envelope (each recipients[].sealed entry is a libsodium sealed box over the data key; try each until one opens):

php -r '
$env = json_decode(file_get_contents($argv[1]), true);
$sk  = base64_decode(trim(file_get_contents($argv[2])));
$kp  = sodium_crypto_box_keypair_from_secretkey_and_publickey(
           $sk, sodium_crypto_box_publickey_from_secretkey($sk));
foreach ($env["recipients"] as $r) {
    $key = sodium_crypto_box_seal_open(base64_decode($r["sealed"]), $kp);
    if ($key !== false) { file_put_contents("data_key.txt", $key); exit; }
}
fwrite(STDERR, "no recipient in this envelope opens with that key\n"); exit(1);
' {archive}.keys.json recovery_private.b64

Step 2 — decrypt the archive (the data key is the passphrase; the cipher is recorded in the envelope's cipher field, aes-256-cbc-pbkdf2):

openssl enc -d -aes-256-cbc -pbkdf2 -pass file:data_key.txt \
    -in {archive}.tar.gz.enc -out {archive}.tar.gz

Then shred data_key.txt. The same two steps work with the site key — config/backup_site_key is the base64 of the raw sodium keypair, so in step 1 replace the two keypair lines with $kp = base64_decode(trim(file_get_contents($argv[2]))); — and that is all the platform's own restore path does. A chain restores by decrypting the full plus each incremental with the one data key from the chain manifest's envelope and applying them oldest-first.

Recovery key setup

Setup happens on the Backups page and needs no shell. The panel is rendered by includes/RecoveryKeySetupPanel.php — one class for all four states (unconfigured / invalid / unproven / ready), so every surface that offers the setup offers the same one.

The default path is one screen:

1. Generate. The page mints an X25519 keypair with WebCrypto (recoveryReadiness.generateKeypair()). The private half is shown once, with copy and download, and is never sent anywhere. The public half never appears on screen at all — it rides the API call in step 2.
2. Paste it back, one button. The operator pastes the private key back from wherever they saved it; the page refuses a paste that does not match the generated key, then one button drives two API actions: backup_recovery_save stores the public half (unproven) and returns a challenge sealed to what was actually stored — re-read from the settings table, not echoed from the input — and the browser opens it with the pasted key (X25519 → HKDF-SHA256 → AES-256-GCM) and posts the recovered sentence to backup_recovery_prove.

Generate-in-browser is the one setup path (a browser that cannot do X25519 is told so plainly). A session that dies between save and proof lands on the unproven state, which runs the same challenge ceremony with the saved private key — in the browser, or via escrow_keypair.php unseal. A key generated at the shell can still be installed by POSTing save_recovery_key (the admin_backups handler), which stores it unproven into the same state.

Once a proven key and a scheduled target both exist, the nightly BackupRun task switches itself on (BackupNightly::maybe_activate, called from the setup-completing requests) — nightly backups are not a decision of their own.

escrow_keypair.php is the disaster-recovery tool — it runs on any machine with PHP and libsodium, with no platform around it:

php maintenance_scripts/sysadmin_tools/escrow_keypair.php generate --private-out ~/recovery.key
php maintenance_scripts/sysadmin_tools/escrow_keypair.php unseal   --private ~/recovery.key

The encoding is one contract across all of them: both halves are the raw 32 bytes, base64, one line. tests/backups/recovery_key_encoding_test.php holds it by running the shipped generator and checking libsodium agrees; tests/backups/recovery_one_screen_flow_test.php holds the save → challenge → prove sequence, and tests/backups/backup_nightly_test.php the activation rules.

Replacing a proven key is a rotation, not an edit: backups already made carry keys sealed to the old public key. Pasting over a proven value is refused.

Rotating the key

Rotation is offered where the key lives: the Backups page's Recovery key section, Actions → Rotate key. It walks the same generate-and-verify ceremony as setup — new keypair made in the browser, private half into the password manager, pasted back to prove the stored copy — with the save marked as a deliberate rotation. Three properties make it safe:

• Old backups keep opening. Each chain's data key was sealed at chain start; rotation never touches it. Keep the old private key until every chain sealed to it has been retired.
• Nothing seals to the new key until it is proven. An interrupted rotation leaves the key unproven, and backups refuse to run — loudly — until the ceremony is finished (or run again with a fresh key).
• The next run starts a fresh chain. A chain cannot change recipients mid-life, so the runner ends the current chain when the recovery recipient no longer matches (recovery_rotated) and the new chain seals to the new key.

Only this site ever sets this site's key

backup_recovery_public_key is the key every backup of this site seals to, whoever took it, and its custodian is whoever administers this site. Nothing writes it from outside. Possession is proven here, against a challenge this site issued: maintenance_scripts/sysadmin_tools/set_recovery_key.php reports what this site holds and refuses to write it, and a management job that passes key material is refused by the tool it passes it to.

An empty slot means this site takes no backups, for anybody. Not its own, and not a management node's copies of it — there is no key those could be sealed to, and an unencrypted whole-site archive on somebody else's shelf is not a fallback. Set the key up at Admin → System → Backups: the page generates a keypair in the browser and runs the possession challenge in one pass, needing no management node and no shell. A management node managing this site can see that the slot is empty and say so on its dashboard — see Server Manager — and that is the whole of what it can do about it.

Uploads

Artifacts reach the bucket through S3Signer — hand-rolled SigV4 against any S3-compatible endpoint, so the backup path carries no SDK dependency. An artifact of 1 GiB or less goes up as one signed streamed PUT. Above that, put_file() switches to the multipart API on its own: no setting, no caller involvement. The threshold sits deliberately far below the 5 GB single-PUT cap every provider enforces, so the multipart path is exercised by routine artifacts (a nightly database dump crosses it) rather than only by the oversized archive it exists for.

Parts are 100 MiB: each is read into memory, hashed, and signed with its real payload hash, so the provider verifies every part's bytes against the signature, and a failed part is re-read from disk and retried on the same budget as any other request. One part is also the peak memory cost, sized for the smallest node. CompleteMultipartUpload responses are checked by body, not just status — a provider can answer HTTP 200 with an <Error> document, and that response is retried and then surfaced as a failure, never recorded as a backup. Any failure aborts the multipart upload so no partial object is left claimable; because an abort can itself be lost (the process can die), the bucket should carry a cancel-unfinished-multipart lifecycle rule (B2: cancel unfinished large files after 7 days) as the backstop.

sha256 and bytes are computed from the local file either way; a restore verifies against them and cannot tell how the object was uploaded.

The upload ledger

Every artifact that reaches the bucket is also recorded on the machine that made it, in config/backup-ledger/{profile}.json: the artifact's name relative to its backup directory, its sha256, its size, and when it went up. BackupLedger writes it, from BackupRunner::upload(), at the moment of upload.

• Local — keep M days in /backups (default 7). The local copy is a convenience, not the archive: it lets a restore skip the download, and this window says how long that is worth the disk. Age is per file, so an old chain's early runs go while its recent runs stay, and the emptied chain directory is left for chain retention to retire. A chain's manifest.json and the snapshot beside it are never swept — they are what make the chain extendable, and without either the next run silently starts a fresh full. The sweep also removes the auto_pre_* snapshots a restore leaves behind, which are the size of a full backup. An archive and its envelope are always swept together. 0 means never. With Delete the local copy once uploaded on, a chain run removes its artifacts as soon as they are confirmed offsite instead of waiting out the window.

Restoring

On the machine itself, nothing extra is needed — the envelope sits beside the archive and opens with the site's own key:

bash maintenance_scripts/sysadmin_tools/restore_project.sh {project} /backups/{archive}

From the bucket, with only the recovery key:

php maintenance_scripts/sysadmin_tools/backup_envelope.php open \
    --sidecar {archive}.keys.json --private ~/recovery.key --key-out /tmp/k
bash maintenance_scripts/sysadmin_tools/restore_project.sh {project} {archive} --key-file /tmp/k

restore_project.sh decides whether an archive is encrypted by reading the openssl magic bytes, not the filename, so a renamed archive still restores. It takes --domain to name the domain the restored site is to answer to; without it the site keeps the domain this machine's config already names.

Nothing is saved before a restore. The schema is dropped and replaced, and the state that was there is not kept anywhere. A restore happens because the current state is wrong, so preserving it preserves what the operator has decided to discard — and it kept a full copy of the database, per restore, indefinitely, on a disk sized for backups rather than for regret. The approval an operator answers already says in words that anything written since the archive was taken is gone.

What that gives up, stated plainly: a load that fails part way leaves the schema already replaced and nothing on the machine to put back (RESTORE_LOAD_FAILED). The answer is the archive itself, which is still on the shelf and can be retried, or an earlier one.

The target's PostgreSQL must be at least as new as the source's. A dump carries the syntax of the version that wrote it, so the restore reads that version from the dump header and refuses before replacing the schema when the target is older, reporting RESTORE_SERVER_TOO_OLD with the database untouched. Restoring onto a newer PostgreSQL is ordinary and needs nothing.

A restore lands on an installed site. The config that carries this machine's database password and secret_box_key is never in a backup, so the sequence for new hardware is: install the site, then restore onto it. See Deploy and Upgrade.

Bucket credentials plus the password-manager private key are sufficient to recover from total loss of the machine.

Restoring a managed node from its management node

A managed node is restored over the agent channel, and it takes three steps because they are three genuinely different decisions.