Server Manager

The Server Manager plugin provides a web UI for managing remote Joinery production servers. Operations include status checks, backups, database copies, and applying updates -- all from the admin interface at /admin/server_manager.

The system has two components:

• PHP plugin (plugins/server_manager/) -- admin UI, job creation, command generation
• Go agent (/home/user1/joinery-agent/) -- runs a management node's own job queue, and on a managed node takes work from its plane over the agent channel

Quick Start

1. Install the plugin

The plugin is already in the plugins/ directory. From the admin panel:

1. Go to /admin/admin_plugins
2. Click Actions on "Server Manager" and choose Install
3. Click Actions again and choose Activate

2. Install and start the Go agent

Release channel (how the agent normally arrives and stays current)

The agent ships inside the platform release. Publishing an upgrade bundles a signed agent artifact into public_html/agent_dist/ — core, not this plugin, because every node must receive it and no plugin may arrive as a side effect of a core upgrade (this one is commercial and entitlement-gated). The plugin builds and signs the artifact; the core archive delivers it:

• manifest.json — agent version, the agent commit the binaries were built from (source_commit), and per architecture the artifact filename, its sha256, and an Ed25519 signature over the raw binary
• joinery-agent-linux-amd64.gz / joinery-agent-linux-arm64.gz — the binaries
• joinery-agent.service — the systemd unit

Bundling is the first thing a publish does, before the VERSION file, the archives or the release row, because its outcome decides whether the release happens at all:

• No agent source on this box — the existing artifact carries forward unchanged and the publish proceeds. Publishing never depends on a Go toolchain being present.
• Source version matches the bundle, built from the source's HEAD — agent_dist is left byte-identical. It sits outside every plugin tree, so a rebuild changes no plugin's tree hash and bumps no plugin's version.
• Source version matches the bundle, built from an earlier commit — the source is rebuilt with the bundle's recorded inputs (version, signing key, baked_keys) and compared. The same bytes record HEAD as source_commit; different bytes refuse the publish until the version in main.go is raised. The release statement names HEAD as the agent commit, and utils/verify_release.php rebuilds the binaries from it, so they must be HEAD's.
• Source is newer and the rebuild succeeds — the fresh artifact is bundled, replacing the agent's own files only (AgentDistPublisher::ownFile(): the binaries, the unit, manifest.json); the support bundle and the release statement beside them go on into the new agent_dist, and because this happens before plugin archives are built it is captured in the server_manager archive and its tree hash.
• Source is newer and the rebuild fails — the publish is refused. The build error is printed, and the VERSION file, archives and release row are all left untouched. Shipping here would mean releasing an agent the publisher already knows is out of date, and the resulting fleet has no way to tell.

The converger has a second job: it is the root actor that carries out root requests. The code tree belongs to root and the web server cannot write it, so an upgrade, a plugin or theme install, a docs save — anything that writes code — is queued by the page as a named request and run by the converger as root, with its transcript polled back onto the page that asked. See docs/deploy_and_upgrade.md. On a managed node the agent does the same work through its own jobs; the converger is what a self-hosted box has instead.

First install is handled by the core installer maintenance_scripts/install_tools/install_agent.sh, which runs at every root moment — site install, code upgrade, container start, the node-detail Run Plugin Installers action, and the host converger's timer on a box whose upgrades run from the browser (docs/deploy_and_upgrade.md). It installs the bundled binary, writes the env file with the right JOINERY_CONFIG, and sets up systemd or cron supervision automatically.

The installer is core rather than a plugin's, and runs on every Joinery instance: the agent does a machine's own backups, upgrades and health checks, and only a management node has server_manager turned on. The artifact stays in this plugin's tree because this plugin builds and signs it, and it reaches every node regardless — the plugin is included_in_publish and receives_upgrades, both independent of whether it is active there.

The binary lands on every deployment. Installing is not running: the artifact is converged at each root moment regardless of the switch, so a machine that is switched on later starts a service that is already there rather than fetching, decompressing and verifying one at that moment.

Whether it runs is one setting, agent_enabled, which ships off and is read fresh at each root moment: on starts the agent and sets up its supervision, off stops it and takes the supervision away — the cron keepalive included, or stopping would last a minute. The agent's identity survives an off, so turning it back on resumes the same pairing. A database the installer cannot reach leaves the machine untouched rather than being read as off.

Three ways to set it, all writing the same setting:

• the machine's own Admin → System → Management Node page, which also says what is still needed for it to take effect
• php utils/agent_control.php --on (also --off, --join=URL, --leave, --status) on the machine
• from a management node, the node detail Agent Channel panel's Turn on the agent over SSH, which switches it on, runs the installer, and has the node ask to join — the fleet path, available only while SSH is, and retired with it at the Phase 3 cutover
• install.sh --enable-agent at install time, with --management-node=URL to have that agent ask to join the named plane in the same step. A site this management node provisions passes both automatically, so a node it builds comes up running its agent and asking to join — the one case where whether the machine should run one is already answered. The join is still a request an operator approves here

A web request has no root, so flipping the setting does not itself start the agent — the next container start or upgrade does, or sudo bash {site root}/maintenance_scripts/install_tools/install_agent.sh {sitename} does it immediately. The Management Node page says so, and prints that command when the switch is on and no binary has reached the machine yet.

Every later version change is handled by the agent itself. Between jobs, the agent compares its own version with the bundled manifest. When they differ, it decompresses the artifact, checks the sha256, verifies the Ed25519 signature against the public key embedded in its binary, keeps the current binary as .bak, renames the new one into place, and exits cleanly for its supervisor to restart. The signature check is the security boundary: the site tree is writable by the web user while the agent runs as root, so the agent never installs anything the publisher did not sign. An artifact that fails verification is refused, logged under a === Self-update === header, and not retried until the manifest changes. The update check and the poll loop share one job lock, taken before a job is claimed: a process holding the lock is either running a job or about to exit for a swap, and in both cases the poll skips its tick, so the plane never hands a job to a process that will not live to report it.

The public log. An agent that holds a release statement key and a log checkpoint key installs only a binary whose release is in Sigstore's public log. Before fetching the binary it reads the release's RELEASE_STATEMENT — agent_dist/RELEASE_STATEMENT on a machine with a site tree, the artifact kind agent_statement over the channel otherwise — and checks it offline: the statement's signature against a statement key it holds, the log entry bound to that statement, the checkpoint against the log's key, the inclusion proof, and that the statement records this binary's sha256 under agent/<platform>. The checks are releaselog.go, the same four includes/TransparencyProof.php makes. A binary that fails any of them is refused with the update state unlogged and not retried until the release changes. The keys are written to the key files before the swap, each file replaced whole by a rename; if they cannot be written, the binary is not installed and the next check tries again. The keys held are the ones compiled into the binary (-X main.releaseStatementKeysB64, -X main.releaseLogKeysB64, set by AgentDistPublisher from release_keys/ and recorded in the manifest as baked_keys) plus /etc/joinery-agent/release_statement_keys and /etc/joinery-agent/transparency_log_keys, which are read only when root-owned and not writable by group or others. The statement's key_chain is walked in memory from those keys; before a swap the install step appends every key held and every key the chain proved to the two files, and nothing removes them — so once a machine has held the keys, every later binary must be logged, including one built without keys. An agent holding no keys updates on the signature alone. On a site node the agent's requirement and the site's own (config/release_log_required) are separate stores and start at different moments: the site's when its converger first writes keys from a bundle, the agent's when it first runs a binary built with keys. The agent never reads the site's config/ key files, because the tree's owner can write there on some machines, and a key added is trust added. Every field of a statement is read by its exact name, as PHP reads it, so the agent and the PHP readers never see different values in one statement.

Before the swap, the agent runs the verified new binary with --version and refuses one that does not answer with the version the manifest names (a binary built for another machine, or one that dies before main), as it refuses a bad signature. After the swap the new version is on trial: .pending beside the binary names it with the count of its starts. It is trusted, and the .bak deleted, once a process running it has kept running for three minutes (trialUptime). A start that finds the previous start of the same version still on trial (it died first: a crash, an out-of-memory kill) puts the .bak back, records the version in .rejected and exits, the supervisor starts the previous agent, and that version is never reinstalled; the next release supersedes the rejection. A deliberate stop during the trial (the operator's switch, a signal, a restart a job asked for, a move, a leave) starts the trial afresh rather than counting against it. No newer version is installed over one still on trial, so the .bak is always a binary the machine has seen run. Before every start, the systemd unit's ExecStartPre and the cron keepalive run the same start guard: while a swap is on trial with a .bak kept, a binary that cannot answer --version is replaced by the .bak and marked rejected. That covers a binary that never reaches the agent's own watchdog in main. The unit itself is converged from the release's agent_dist on a site, and on a machine with no site from the copy built into the binary (the same install/joinery-agent.service), which the agent puts back within a minute of it going missing or being edited.

The dashboard's Agent Status bar surfaces all of this from the heartbeat row (ahb_bundled_version, ahb_update_state): a pending update, a refused (verification-failed) artifact, a binary refused because its release is not in the public log (unlogged), a rolled-back version, or an agent built without an update key.

A paired agent also reports where its self-update stands on every poll (1.65.0 and later): its verdict on the version on offer (update_state, a closed set: none when nothing is on offer, current, update_pending, fetch_failed, verify_failed, unlogged, version_rejected, unsigned_build, no_binary) and that version (update_offered), stored as mgn_agent_update_state and mgn_agent_update_offered. It sends them only once a check has concluded since it started, and an absent report leaves the last one standing. A refusal raises the plane:agent_update_refused incident. A siteless machine holds its support bundle to the same bar: with the log keys, it installs a bundle only when the release statement in the public log records its bytes (artifact support_bundle), keeps the keys the statement proves first, and reports its verdict on every poll as bundle_state (current, unlogged, verify_failed), stored as mgn_agent_bundle_state; a refusal raises plane:agent_bundle_refused. Publish writes the binaries and the bundle before the statement that records them, so an unlogged verdict, on either, is reported only once it has stood 15 minutes; an unlogged bundle is tried again whenever the statement beside it changes.

Manual install (bootstrap fallback)

For a management node that has no bundled artifact yet, build and install by hand:

cd /home/user1/joinery-agent
PUBKEY=$(cat /var/www/html/joinerytest/config/agent_signing_key.pub) make release
sudo bash joinery-agent-installer.sh --verbose [--config /path/to/Globalvars_site.php]

make release compiles the binary (passing PUBKEY bakes in the update-verification key so the manual build can still self-update later; STATEMENT_KEYS and LOG_KEYS bake in the release-log keys, in the forms the publisher passes) and packages it into joinery-agent-installer.sh, a self-extracting script that handles both fresh installs and upgrades with automatic rollback if the new version fails to start.

The installer detects the host's supervision capability:

• systemd hosts: installs /etc/systemd/system/joinery-agent.service; start with systemctl start joinery-agent.
• No systemd (Docker containers, minimal hosts): installs /usr/local/bin/joinery-agent-supervise plus /etc/cron.d/joinery-agent (@reboot + a once-a-minute keepalive) and starts the agent immediately. Logs go to /var/log/joinery-agent.log.

Both modes create:

• /usr/local/bin/joinery-agent — the binary
• /etc/joinery-agent/joinery-agent.env — configuration (from example, first install only)

Every management node needs a live agent. Jobs sit pending until an agent claims them — the node's own paired agent for everything but install_node, which is the one bootstrap the plane runs itself (see Customer-Cloud Fulfillment). The Server Manager → Provisioning page shows an agent heartbeat badge as requirement #1.

Configure (usually not needed)

The agent reads database credentials directly from Globalvars_site.php — no manual configuration required on a standard Joinery install.

The default config path is /var/www/html/joinerytest/config/Globalvars_site.php. If your install is at a different path, set it in the env file:

sudo nano /etc/joinery-agent/joinery-agent.env
# Set: JOINERY_CONFIG=/var/www/html/mysite/config/Globalvars_site.php

Other optional settings in the env file:

Start

sudo systemctl start joinery-agent
sudo systemctl status joinery-agent

The dashboard at /admin/server_manager should now show Agent Status: Online.

If anything is wrong, the agent logs to the systemd journal (systemd hosts) or /var/log/joinery-agent.log (cron-supervised hosts):

journalctl -u joinery-agent -f      # systemd
tail -f /var/log/joinery-agent.log  # cron supervision

Common startup errors are self-explanatory — missing DB_NAME, wrong password, or plugin tables not installed. Each error message tells you exactly what to fix.

Upgrade

Agents upgrade themselves from the bundled artifact after each platform release lands (see Release channel above). The manual-install path also accepts upgrades: re-running the generated installer stops the service, swaps the binary, restarts, and rolls back automatically if the new version fails to start.

3. Add managed nodes

Go to /admin/server_manager/node_add (or click Add Node on the dashboard). There are two ways to add nodes:

From the node

A site this management node did not install is enrolled from the node's own Admin → System → Management Node page: the admin there enters this management node's URL, the node's agent generates a keypair and asks to join, and the request appears on the node's API keys tab here for approval after a fingerprint comparison. Add the node's record below first so the join has something to be approved against. The plane never needs a shell on someone else's machine — there is no discovery scan (see specs/ssh_single_bootstrap.md).

A machine this management node provisioned

A cloud instance created from Remote Install runs two agents on a docker box: the site's, inside the container, and the host's own, beside it. Both ask to join on their own, named <slug> and <slug>-host. The instance is known by both the addresses the provider reported for it (IPv4 and IPv6), so a join from either family is recognised as that machine. The site's join is approved on the provision's node, from its API Keys tab, where the provider is asked first to confirm the instance is running at the join's address. The host's join is approved from the dashboard banner: the same provider check runs, then the host node is made at the instance's IPv4 and named on the placement record, which is what routes host-scope work (certificates, site removal) to it.

Rejecting a join is reversible for a day. The dashboard lists joins rejected in the last day with a Reopen button; the machine keeps its key and asks again every five minutes, so a reopened request is answered by the same fingerprint the operator already compared and can then be approved like any other. joinery-agent leave on the machine is what discards the key.

Retiring the install password is proven by the machine refusing it. A retirement job that finds the machine already refusing the password before its script runs completes as retired: the refusal is the proof, so a record that still says "held" catches up when the job is re-run.

Manual

Fill in the form fields directly:

Admin Pages

All pages are at /admin/server_manager/.... Each page's @joinery-admin header sets who may open it (Permissions): the dashboard, node detail and the job pages at 7, which run the look jobs (check status, logs, file head, page probe and the other reports) and read their redacted output; the other pages at 8; the backup target pages (keys), and the pages that add hosts, nodes or machines, provision a server, or manage domains, transfers or service tenants, at 10. What a POST on a page does needs its own level (SmActionLevels): 8 to change a node (restart, update, publish, run a backup, cancel a job, triage an incident), 10 for anything destructive or key-changing (restore, delete, purge, move or copy a site, reboot, root access, a join's or an escrow's trust, a credential, the data root's encryption, deleting a release). Below 8, node detail shows only its reading tabs. The API actions declare the same levels.

The UI is organized around a dashboard + node detail pattern. The dashboard shows the fleet overview; clicking a node opens a tabbed detail page with all operations for that node.

Node Detail Tabs

The node detail page (/admin/server_manager/node_detail?mgn_managed_node_id=N&tab=...) has seven tabs:

Running a command on a node

There is no way to. A management node holds no mechanism for running an instruction it composed at runtime on a managed node — no Console tab, no node_exec.php, no run_command job (decision A1).

What a node accepts instead is a primitive: a name it looks up in a vocabulary compiled into its own agent, with parameters validated against a declared shape. A job names an operation; it never carries a command. A node refuses anything outside that vocabulary whatever this plane asks, so what the fleet can be made to do is bounded by what was built and signed, not by what someone can compose.

Work with no primitive yet is done over the operator's own SSH key. That is deliberate and it costs something real: those sessions are not recorded, where a console run produced a job row with its command, operator, output and exit status. The audit trail was given up knowingly, because a recorded path that can run anything is still a path that can run anything, and a compromised plane would have used it. sshd stays on across the current fleet (A11), so this is the working route until the vocabulary covers the operation.

Dashboard Features

The dashboard shows:

• Agent Status -- online/offline indicator with version and last heartbeat time
• Managed Nodes -- rows with health-based status dots (green=healthy, yellow=warning, red=problem, gray=no data), grouped by what a node is. A host group is a Docker box: its header shows the placement record (mgh_managed_hosts — address, container count against mgh_max_sites, whether automated provisioning may place customer containers on it), its first row is the box's own agent node (badged host agent, never counted as a site), and the rows under it are the containers placed there. Every other node — a bare-metal site, a relay, a DNS box, the management node itself — is a machine in its own right and is listed flat under Machines; a machine has no host record because its box identity is its node. A node placed on a host record that has since been deleted is listed as a machine, not dropped.
• Publish Upgrade -- build upgrade archives from management node source code, as a job of the management node's own agent
• Recent Jobs -- latest 20 jobs across all nodes

• Red: Last check failed, disk > 90%, or PostgreSQL not accepting connections
• Yellow: Disk > 80% or load average > 5
• Green: All metrics healthy
• Gray: Never checked or no data

Edge versus direct

Compare edge and direct on a node's Overview answers one question: is the site slow, or is the path to it slow? This management node times the same requests two ways at the same moment: through the site's public name (its edge, such as Cloudflare or a relay) and straight at the machine's own addresses (mgn_host, and each public address in mgn_addresses, IPv4 and IPv6, which the agent reports at its join and in each escrow state post). No agent is involved; EdgeOriginProbe runs inside the request.

Each of up to 20 rounds sends, at once down every path, a GET of /login and a POST of {} to /api/v1/action/vault_heartbeat, which refuses it without a session; the POST is there because a request body that never arrives is a stall of its own. A request that takes 5 s or more, or never answers within 10 s, is a stall, and no round starts after 45 s, so a run finishes inside the 100 s the edge in front of this management node allows. The direct legs verify neither the certificate's name nor its chain (an origin behind an edge may present a Cloudflare Origin CA certificate or Apache's fallback one); the result names the certificate each origin presented. Requests identify themselves as JoineryEdgeProbe/1, which sites never count as visitors.

The completed job (edge_vs_origin) carries each path's requests, stalls, 50th and 95th percentile, statuses, errors and 429s, and one verdict line: Slow through the edge, fast direct, Slow on the server, or No problem seen, with a note when the site's public name resolves to the machine itself and both legs took one path. The POST legs carry no credential, so they count against the site's own per-address limit and land as no_credential rows in its request log; a node is compared at most 10 times an hour, and a 429 is reported as the site's limit, not as a stall. Re-run on the job page compares again.

The agent channel

A node's agent polls its management node over an outbound HTTPS connection, takes one job at a time, and posts the result back. Nothing has to reach in: a node behind NAT or Cloudflare works the same as one with a public address, and the poll itself is how the plane knows the node is alive.

A job names an operation; it never carries a command. The payload is {primitive, params} — a name the agent looks up in a vocabulary compiled into its own binary, plus parameters validated against the bounds that primitive declares. A name the agent does not have, a parameter it did not declare, or an operation class the node does not accept is refused on the node, with the reason recorded and reported back, whatever the plane asked for.

Primitives are grouped into three classes, and a node's acceptance policy is set per class:

The policy lives at /etc/joinery-agent/policy.json, root-owned, outside the web tree. A missing file means the shipped fleet-wide policy above. A file that is not root-owned, or is writable by group or other, is refused outright and the node accepts nothing until a human fixes it — trusting a file the web user could have written would defeat the only thing the file is for.

Connecting a node (the join)

Enrollment is node-initiated and shares no secret. On the node, Admin → System → Management Node: the site's admin enters the management node's URL — just an address. The node's root agent generates an Ed25519 keypair, keeps the private half at /etc/joinery-agent/node_identity.json (mode 0600, root-owned), and sends a join request carrying only the public half, a claimed name and, when the machine has a site, the site's public_html path. The node's page then shows the key's short fingerprint and waits.

On the management node, the request appears on the node's Detail → API Keys tab: claimed name, source address, and the same fingerprint. Approving is superadmin-only and should happen only if the fingerprint matches the node's own page exactly — the name and address are claims anyone could make; the fingerprint comparison is the entire security of the introduction. Approval binds the public key to the node record, and fills an empty web root from the one the join named; the agent picks it up on its next check and both panels flip to Connected. After that, each status check fills an empty web root and an empty site address the same way, from what the agent reads in the site's own config: the site address becomes https:// plus the config's domain (webDir). A set value is never replaced, and an address another node already holds (other than a copy's source) or this management node's own is left empty and logged.

The management node stores only the public half, so there is no credential on it that could act as the node — and enrollment adds nothing to steal: no token, no code, nothing copied by a human. A wrong approval is visible (the tab stamps the connection time, agent version, and last poll) and severable with the Disconnect button, which forgets the key and drops the node back to API/SSH routing.

Requests are rate-limited, expire after an hour (the agent renews its own while it waits), and are capped in number. A rejected request retires its keypair — the agent introduces itself with a fresh key on the next ask.

The web tier's only involvement on the node is a handoff through three managed settings: agent_join_request (the URL the admin asked for), agent_join_state (the agent's progress, which the page renders), and agent_leave_request (the admin asked to disconnect). None ever holds a credential.

Routing

A connected agent is routed to — approving the join is the routing decision, and there is no further switch. An operation with a primitive implementation runs on the node's own agent, and only there: a node without a paired agent is refused such a job at build time, with the fix (pair the node) in the message. SSH exists only for the bootstrap pair — install_node and retire_install_password, both run plane-side by InstallJobExecutor over a provision's sealed root password. Disconnecting the agent therefore stops a node's operations until it pairs again.

Either side can end the pairing, and neither needs the other's cooperation. This plane's Disconnect button forgets the node's key. The node's own Management Node page has a Disconnect too: its agent finishes any running job, sends one signed goodbye to /api/v1/agent/leave (so this plane forgets the key immediately), deletes its identity, and then takes no work at all until it pairs again — and it leaves even when the goodbye cannot be delivered, in which case this plane just sees the agent go silent until someone disconnects the node here as well. Both endings run through AgentChannelEndpoint::forgetAgent(), so they cannot drift apart.

The node's Management Node page shows the connection in use however it began (from that page, joinery-agent join, a site copy or move_to_plane): a connected agent writes it into agent_join_state and rewrites it whenever the page's view drifts, and clears a join request recorded while it is connected, which it would otherwise ignore.

Moving a machine to another management node. A site moves from its own Management Node page: Disconnect, then Connect to the other address. A machine with no site (a Docker host) has no page, so this plane asks for it: the API Keys tab's Move to another management node takes the other management node's address and sends move_to_plane (agent 1.67.0; JobCommandBuilder::build_move_to_plane). The agent stages a fresh keypair for that management node and files the join there under this node's slug, so an approval there gives it the same slug and its backups keep their folder name. That holds when the machine moves to its own management node too: the plane joining itself shows as its own site's host, but takes its slug from the claimed name (AgentChannelEndpoint::adoptJoin). The job's result names the fingerprint to compare before approving it there. The machine stays connected here until then, so an ask that is rejected or never answered loses nothing; once it is approved, the agent stores the new credential, sends this plane the same signed goodbye a Disconnect sends, and restarts onto the new management node. joinery-agent leave on the machine cancels a move in flight.

After a goodbye. The goodbye also records, on the node, when the agent left (mgn_agent_left_time) and, when it followed a move_to_plane this plane sent since the pairing began, the management node it went to (mgn_agent_left_to, from that job's management_node; AgentChannelEndpoint::recordLeave). From then on ManagedNode::agent_moved_away() answers for the row: the node page says 'Its agent moved to … on …' above every tab and that nothing here reaches the machine; Connect agent, the DNS box and Reverse DNS are not offered (Set Reverse DNS refuses); and fleet backups, patch runs and Deploy to all managed nodes leave it out without listing it. Its uptime check from outside keeps running when switched on, so a row kept only to watch a site that manages itself still does that job. A new pairing clears both fields. A plane-side Disconnect is not a goodbye and records neither.

Operations cross one at a time. An operation has crossed when JobCommandBuilder has a build_<op>_primitive method for it; transports_for() discovers that by reflection, and ManagementJob::createFromBuild() stores the right shape without any caller knowing which transport ran.

A primitive is routed to a node only when that node says it has it. Every claim carries the agent's own list of the primitives its binary compiled in, and the plane stores it (mgn_agent_primitives). Routing consults that list: an operation missing from it is not sent to the agent — the builder refuses at build time (or, for check_status, falls to the probe transport). The node's own account is the only one that is not a guess — a version number says which release a machine runs, and only the machine says what that release compiled into it.

The version floor. AgentVocabulary::FLOOR is the oldest agent this management node supports (1.43.0). A node whose agent is below it — or reports no version, or no vocabulary at all — is offered apply_update and nothing else, which is how it gets back above the floor; the node list flags it. The floor is a constant that rises in a release, never a setting: the code that spoke to older agents is deleted each time it rises, so there is nothing below it for the plane to say. At or above the floor, the node's reported words decide, and an empty report is no evidence of any word.

Word contracts never change. A word's name, parameters and meaning are fixed once it ships; its output may gain fields and never loses or redefines one, and the plane reads a missing field as not reported (null), never as zero or no. A change to what a word takes or does is a new word with a new name. That is what makes routing by name sufficient on a fleet running several agent versions.

A feature declares the words it needs. A feature built from several words asks AgentVocabulary::missing_words($node, [...]) once, and on a node that lacks any of them shows the one standard state — Needs a newer agent; update this node, naming the missing words and, below the floor, the floor (needs_newer_agent_text / needs_newer_agent_html) — never an error, a blank, or a half-rendered card. Every builder refuses a missing word with the same sentence.

Both directions of a version gap. A claim field this plane does not know is set aside unread (AgentChannelEndpoint::known_claim_fields) and the rest of the claim is taken, so an agent newer than its management node keeps reporting its vocabulary; a word this plane has no builder for is stored in the vocabulary and never routed. Against a management node that still refuses an unknown claim field, the agent drops only the field the refusal names and claims again. tests/agent_version_spread_test.php holds a recorded older agent's poll and a newer agent's poll against this plane; the agent's vocabulary_test.go holds the other side.

Version spread on the node list. Each node row shows its agent version, how many releases it is behind the agent this management node ships (agent_dist/manifest.json), and a below minimum badge under the floor. A staged rollout (/admin/server_manager/rollouts) is how the spread is closed.

The management node is a node like any other. It pairs to its own site, and the work it does for itself — publishing a release — reaches it the same way every other operation reaches every other machine: as a primitive its own agent looks up in its own vocabulary. There is no second job source anywhere in the fleet. A row written into mjb_management_jobs by hand executes nothing, on any machine.

Recipes: what the agent does on its own clock

Besides the words the plane may ask for, the agent carries recipes: fixed sentences compiled into its recipes package, each one check, repair, verify, retry, hold, escalate for one aspect of the host. A recipe takes no parameters and is registered like a primitive, with its own pinned registry test, so a recipe cannot arrive in a build without a human naming it. The loop runs on every agent from 1.27.0, paired or not — a self-hosted box with no management node is exactly the machine that has nobody else to repair it — and nothing on the wire names, starts, stops or arms one.

The loop (recipes/loop.go): every ten minutes each recipe's check runs. A check answers pass, fail or unknown; unknown (a container, a box without systemd, a script the node cannot verify) never repairs and never counts. A repair runs only after two consecutive failing ticks, under the same job lock a plane job holds (a tick that cannot take it writes "a job is running" and tries next tick) and the same job marker (/etc/joinery-agent/job-running, pid first, then recipe <name>). Three attempts per failing run (the attempts since the check last passed, a repair last verified or the escalation closed), the second ten minutes after the first and the third thirty after the second, each due on the first tick at or after its wait even when the timer lands that tick a little early; a fourth is an escalation — one open per recipe, appended to on every further failing tick, closed when the check passes — during which the recipe checks and never repairs. An operator working on that aspect sets a hold marker, touch /etc/joinery-agent/hold/<recipe>: the recipe records that it is held on every failing tick and repairs nothing until the marker goes. The hold directory must be root's and writable by root alone, or the marker is ignored and the ledger says so.

Three recipes are compiled in. fail2ban: the unit is active with at least one jail, or host_converge runs host_housekeeping.sh. agent_supervision: something would restart the agent if it stopped (agent_report: systemd supervising this process with a restarting unit, or the cron keepalive installed and switched on), and something starts it after a reboot (a unit that restarts the agent but is not enabled fails), or agent_converge runs install_agent.sh, which writes the supervision and, under the job marker an attempt holds, restarts nothing. The host timer's daily converge already ran that installer, so a lost unit file or keepalive was back within a day; the recipe makes it minutes. An agent that a person started by hand on a systemd box is genuinely unsupervised and the installer cannot fix that without restarting it, so three attempts open a case and a person restarts the agent. disk_headroom is check-only (Recipe.NoRepair): the site filesystem has at least 10% and at least 5 GiB available, read from host_report's disk.avail_bytes and disk.total_bytes; unknown when either is unknown. It has no repair — there is no safe automatic answer to a full disk — so the loop opens a case on the first failing check, spends no attempts, and closes the case when the check passes. It runs on every node, paired or not, which is how a self-hosted box with no management node is warned about its own disk. Its rule has no slope and no history on purpose: the node holds no series. Register permits NoRepair only with no repair word and no repair function, and the registry test pins which recipes are check-only.

Four more are compiled in, and each repairs with a value the node derived for itself (Recipe.NodeDerived): a recipe may call a word that takes a parameter only when the value comes from the node's own answer — its own expected-units list, its own container list, its own vhost — never from the wire or a file, and the word validates the value against its own compiled set whatever the recipe passes. The registry pins which recipes are node-derived. service_health: PostgreSQL, PHP-FPM and Apache answer, not merely run — host_report's answers (a request for the site's own name through Apache that must come back through PHP, and pg_isready), beside the unit states; a unit systemd has given up on, or one that runs and does not answer, is restarted with restart_unit, the first failing in dependency order (PostgreSQL, then PHP-FPM, then Apache), and an absent unit (a database on another host) is not this machine's to restart. container_health: on a Docker host, each site container runs and its site answers through PHP on its published port, or restart_container restarts the first that does not. A container held stopped (held in host_report's containers: hold_container's mark, /etc/joinery/sites/<site>/held) is left alone, and restart_container refuses it. Neither service_health nor container_health repairs while the data root is stopped by hand (host_report's data_root_stopped, agent 1.81.1): what is on it is down on purpose, and restarting a service on it would start the data root again. certificate_expiry (hourly): every name the site vhost serves has a certificate with at least 14 days left, read over this machine's own connection with SNI (served_certificates); fewer means certbot's own timer failed, and provision_certificate re-issues the name with the fewest days, the domain taken from the node's own vhost. mail_gateway (host-scoped): while a site on the machine has mail apps on, the mail apps gateway (joinery-mail-gateway, plugins/mail_apps) runs and greets a mail program on 993 and 465 under each site's mail name (host_report's mail_gateway); a stopped gateway, or one that runs and no longer greets, is restarted with restart_unit. systemd already restarts a gateway that exits. The unit is a constant in the recipe, never read from the report. The agent itself is never one of service_health's units: its supervision is agent_supervision.

release_files (release-scoped: a site installed from a release, never a source checkout) is the deployment files' own recipe: the five files an upgrade starts from (utils/upgrade.php and its four companions) match the signed release, read from upgrade_preflight's manifest and files facts alone (its disk, lock, VERSION and theme checks are a rollout's business and never fail this recipe), or repair_release_files puts each differing one back from the agent's kept copy or from this management node, asked by the hash the manifest lists. An unusable manifest is unknown here, never a failure: that is the manifest healer's case, and a repair would be refused for the same reason. See Script trust and the manifest for the state it clears.

A recipe has a scope. release_files is release-scoped: its subject is a signed release's own files, and a site that is a source checkout (a .git at the site root or the web root: the publishing site, a developer's box) has a tree that is the developer's, legitimately different from the last published manifest between publishes, so the recipe never ticks there, the claim reports it not-applicable, and both restore_release_file and repair_release_files refuse on such a site naming the checkout. fail2ban is host-scoped: its subject is the machine's units and jails, and an agent inside a container cannot see that machine. agent_supervision is site-scoped: its repair is an installer that lives in a site tree and is not in the support bundle a machine with no site runs from. Where a recipe does not apply the loop does not tick it: one journal line and one ledger line (not_applicable, with the reason) at start, and the claim reports the recipe as name:not-applicable, which the node page's Agent box shows as not here. The host's own agent watches the host.

Two ledgers, both JSON lines, one per recipe: /etc/joinery-agent/ledger/<recipe>.jsonl is root-only and is the one the loop reads back (the attempts of the current failing run, the open escalation); an attempt is written when it starts and its outcome when it ends, so a repair that kills the agent is still a counted, failed attempt on the next start. {site}/cache/recipes/<recipe>.jsonl is the same lines, readable by the site, written and never read. Events: check, attempt, outcome (repaired, failed, report-only, interrupted), held, hold_ignored, busy, escalation, escalation_note, escalation_closed. A check line is written on every failing tick and whenever the verdict changes; a pass after a pass, or an unknown after an unknown, writes nothing, so a healthy recipe's ledger is its restarts and a recipe that cannot answer does not fill its ledger with the same answer.

Armed or report-only is a compiled constant in the agent (recipes.ReportOnly, false: the loop is armed and repairs). While report-only is on, an attempt is ledgered with the outcome report-only and runs nothing; the two-tick rule, the backoff, the budget and the escalation count exactly as if it had run, so the ledger shows the path the armed loop would take. Arming or disarming is a release, never a setting. The fail2ban repair counts as run when the converge transcript carries the runner's ok line for host_housekeeping.sh and not its failure line; on a machine with no site the runner runs the whole host set and says more after that line, so the line is looked for, never required last.

The recipes an agent runs, each with its mode and, once its check has run, what the check last said, travel in every claim (recipes: name:armed:fail, name:report-only:pass, name:not-applicable, comma-separated; the verdict is pass, fail or unknown), are normalised on intake like the vocabulary (AgentChannelEndpoint::normalised_recipes, which drops an entry whose verdict is outside that set and drops the verdict from a not-applicable recipe, since one is never checked) and stored in mgn_agent_recipes. The Agent box on the node page lists them, one line each, so a person can see whether a node acts and whether its check passes. A failing check is the step before a case: an armed recipe is repairing it on its own clock, and if it cannot, the case it opens is the incident. A report-only escalation opens a case just the same.

The case

When a recipe gives up — three attempts in an hour and the check still fails — its escalation is a case: the same node-minted id, the same open/closed state, given a body and a delivery. The body (recipes/case.go) is composed once, when the escalation opens: the loop's mode, the attempts that spent the budget (time, word, outcome, bounded detail), a fresh host_report, the vocabulary and the recipe list, every field capped before it leaves. The node closes the case: the recipe's check passing again is the one close, and the plane never tells the node a fault is gone. A failing tick while a case is open is a note appended by id, never a second case; a new case cannot open until the old one closed.

On a paired node the case rides the claim in cases, a JSON object keyed by source (recipe:fail2ban; the unexplained-root classifier's own source lands in the same field once it is built): for each source the most recent case, open or closed, as a summary on every claim, plus the body until a claim carrying it succeeds. A closed summary keeps riding until the next case replaces it, so a close is never lost to a plane that was not listening. cases is one of the extras: a plane that predates it refuses the field and the agent drops every extra, as for recipes.

Intake (AgentChannelEndpoint::intake_cases) caps and re-validates every field — the field as JSON at MAX_CASES_BYTES, at most MAX_CASES_PER_CLAIM entries, free text at MAX_CASE_TEXT, attempts at MAX_CASE_ATTEMPTS with details at MAX_CASE_ATTEMPT_DETAIL, the host report through sanitise_host_report, the lists through the claim's own normalisers, times only as RFC 3339 UTC — and stores the result in inc_incident_records (IncidentRecord, node-scoped: the case store settled question Q3 names). A new id is stored; a known id is appended to (note count, newest note, a body when one rides); a close the node reports is recorded; a closed id never reopens; a source must be a recipe the node reports or a source the plane knows; at most one open case per source per node (a higher id arriving open closes the older one with a note that its close was not heard, a lower id is refused) and at most MAX_OPEN_CASES_PER_NODE open per node. A refusal is logged and costs the claim nothing else.

A stored case is an incident (see Incidents): intake gives it its plain title (IncidentTitles, the plane's own words for the recipe; the node's reason is shown under it as the node's account), its severity (warning), triage new, and its timeline: an opened event at the node's own opening time, and a cleared event when the node reports the close (or when a newer case shows the old close went unheard). A case first heard already closed still opened and still needs a look. Nothing on this side ever closes a case. A case rides every claim, and a claim that changes nothing writes nothing: an open case is re-stamped as still seen at most every ten minutes (inc_last_seen_time), and a closed summary, which rides until the next escalation replaces it, is never written again. A note is taken when the node's count is higher or its note time is newer, so a node whose ledger was trimmed and whose agent restarted (its count starts lower) still lands its newest note; the stored count never goes down.

On the node itself the agent also writes a rendered copy of the case outward, {site}/cache/recipes/<recipe>.case.json beside the outward ledger, written whole and never read back, carrying the case, its body and its delivery (local, or management node when a plane is polling). The site's admin header shows an open one as a core notice (RecipeCaseNotice) worded "as reported by the agent's ledger", on a paired node too (saying the management node has it). The record carries when the agent rendered it, and the agent renders an open case again on every failing tick, so the notice says when the agent last reported the case and, once that is more than a day old, that the record may be out of date and the agent should be checked. The hourly RecipeCaseMail task is the unpaired path only: while a case whose delivery is local is open and the agent has rendered it within the day, it sends the superadmins one plain-text mail per recipe per day — no markup, no link, the last send per recipe kept in the recipe_case_mail_log setting. A record the agent has not rendered for a day is not mailed until it renders it again, so a stopped agent's frozen record is not a daily mail forever; and a case a management node has is never mailed from the node, because the plane's Incidents page and header line already carry it. The web user can forge that file; what a forgery buys is a lie to the admin and one mail, and nothing root acts on.

Log access: what the plane may read of a node's logs

The log words — and file_head, which reads host configuration — are gated on the node by its owner. agent_log_access is a managed setting on the node, off until its owner turns it on, switched from the setup wizard's Beta permissions step or the node's own Management Node page (/admin/admin_management_node), where what it grants is explained in the owner's words. The agent reads it with the same reader it reads agent_enabled with, projects it into a root-owned marker so the file word still answers when the node's database is down, and refuses both words with this site's owner has not allowed log access (agent_log_access is off) while it is off — whatever the plane says. The node reports the switch on every poll as log_access (on or off), stored in mgn_agent_log_access, so the plane's Logs disclosure shows the reason before a job is queued. A node that was already connected when the switch arrived shows its administrators one notice (AgentLogAccessNotice) until they acknowledge it or use the switch.

The node also reports its owner's answer on emergency root access for the beta (the managed setting beta_root_access, agent 1.94.0+) on every poll as root_access (on or off), stored in mgn_agent_root_access when it is one of those and differs. A poll whose database does not answer sends nothing, and the last answer stands. Nothing acts on off by itself: the holder of the box's kept password sees it on the dashboard and retires it.

Everything the words return is redacted on the node by the agent's redact package before it leaves: credential values by the key names SmSecretRedactor masks (the two lists are pinned equal by tests/unit/agent_redactor_parity_test.php), in the quoted shapes and in the assignment shape in any case (PGPASSWORD=..., password=..., api_key=...; a lowercase name ending in _key is a flag or a column and stays readable), the local part of email addresses (the domain is kept, since a mail diagnosis needs it), IP addresses, and long opaque tokens. The plane's SmSecretRedactor is the second pass, at display time, as for every transcript. Member addresses, submitted forms and raw payloads are not returned at all: the table word reads a compiled column list that leaves them out.

What a log job returns is recorded as the job's result by JobResultProcessor (process_site_log, process_site_log_count, process_log_table_tail) from the node's envelope, reduced to the keys the job page renders and bounded on intake (text at LOG_EXCERPT_MAX_BYTES, rows at JobCommandBuilder::LOG_MAX_COUNT, columns at LOG_TABLE_MAX_COLUMNS, cells at LOG_TABLE_MAX_CELL); a completed job whose output was not a log envelope records read: false and the job page says so. The job page renders a site_log result as a log box and a log_table_tail result as a table, both through SmSecretRedactor. A unit_journal result is recorded the same way (process_unit_journal) and rendered as the unit's verdict — state, result, exit status, last run — above its journal lines; its lines age out on the same retention window as a site log's, since they are the same kind of thing (ManagementJob::LOG_EXCERPT_TYPES). A reset_failed_unit result (process_reset_failed_unit) is the unit's state, sub-state and result before and after, and whether systemd accepted the reset; the job page shows both states. A reset the node accepted queues one host_report (unless one is queued or ran in the last minute), so the Machine tile that offered Clear re-measures rather than naming the cleared unit until the next report. A file_head result (process_file_head) is kept like a site log's — name, path, line count, text at the same cap — and ages out on the same window; each line was redacted on the node by its configuration shape (redact.Config: a line whose key names a credential is the key alone, whatever the separator; a secret named later in the line loses its value; addresses in configuration are infrastructure and stay). A disk_usage result (process_disk_usage) is rebuilt as two size tables, every entry reduced to one path and one byte count, both capped at DISK_USAGE_MAX_ENTRIES; nothing in it ages out, because there is nothing in it to age out.

An excerpt does not live on the plane for good. ManagementJob declares a retention rule in the method form: after server_manager_log_excerpt_retention_days (30 by default; 0 keeps everything) the standard sweep blanks the result and transcript of a completed log job and writes {"pruned": true} in its place, so the job row stays as the record that the plane read that node's logs on that day.

Reading refusals

A refused job is a terminal failure like any other, and reads as one everywhere a job status is shown — the message says Refused by the node: and then the node's own reason. The outcome the node actually reported is also recorded on its own (mjb_agent_outcome: completed, failed or refused), so a refusal can be counted rather than found by matching the text of an error message. ManagementJob::refusalCountForNode() is that count, windowed.

This matters more as the vocabulary grows. A node refusing work is a node whose plane is asking for something it should not be, or whose policy has been tightened without the plane noticing — either way it is the number, not the prose, that an alert reads.

A job this plane gives up on records the outcome lost (ManagementJob::OUTCOME_LOST), never one a node can send. The node never reported one, and inventing a verdict for it would make the refusal count untrustworthy in exactly the situation where it is being consulted.

When a claim does not come back

An agent claims a job and then reports. If it never reports — it crashed, the box rebooted, the network went — the job would otherwise sit in running holding that node's concurrency lock. A job is never handed out a second time: whatever the node was doing may well have happened, and running a move, an upgrade or a restore again, unattended, is worse than not knowing. A lost job fails with the outcome lost and a message saying what it did on the node is unknown, and a person decides whether to run it again. It is found lost three ways:

• The agent checks in idle (agent 1.72.0+). An agent claims only while it runs nothing and owes no result, and says idle on that claim; whatever this plane still has as running for the node is failed as lost at once (ManagementJob::markLost). This is what catches an agent or a machine that restarted mid-job.
• Its claim budget runs out (ManagementJob::failStaleClaims, on every poll and every scheduled uptime pass): 15 minutes, or the primitive's own budget (PRIMITIVE_CLAIM_BUDGETS). This covers an agent that never comes back, or one too old to say it is idle.
• An operator marks it failed: Mark as failed on a running job's page, behind a confirm. It frees the node for its next job.

Status facts are kept current on a cadence. Version, certificate, disk and memory facts are measured only by a check_status job. Every scheduled uptime pass queues one for each enabled node whose agent offers the primitive and has no check_status completed inside the last hour (RunNodeUptimeChecks::STATUS_REFRESH_SECONDS, matching the hourly plugin health report), whatever the node's uptime setting — the up/down probe and the facts measure different things. The job table decides, not mgn_last_status_check, which the probe also stamps when it reads a site's health document. A queued or running check_status, or one completed inside the window, counts as cover, so a stale node yields one job per window. host_report rides the same cadence on its own stamp (mgn_last_host_report_time), except that a node whose last report says its data root is being encrypted, decrypted or re-keyed is asked every five minutes (DATA_ROOT_RUN_REFRESH_SECONDS); a machine this management node asked to reboot, until a report carries a new boot id, and an enrolled machine that told this escrow holder its data root opened after its last report (DataRootEscrowSync::opened_since_report(): a site copy's go-back, the provider's restart, a crash) are asked every two minutes (REBOOT_REFRESH_SECONDS), so the node page's 'opened by' is never the boot before; and a data root job that ends queues one at once (JobResultProcessor::report_after_data_root_job()).

Endpoints

POST /api/v1/agent/join, /join_status, /claim, /result, /leave, /quiet, /artifact, /job_status. Join and join_status are unauthenticated by nature — until approval there is no identity to authenticate — and grant nothing; they are rate-limited, validated, and bounded in number. Claim and result are signed with the node's key; the plane verifies against the public half it holds, and selects jobs by the identity the signature proves, never by anything in the request body. Requests are schema-validated, size-capped, and refused for a clock more than five minutes from the plane's. job_status (signed, agent 1.52.0+) answers whether one of the node's own jobs was withdrawn here, cancelled or removed: a node waiting for its operator's approval asks every half minute, and a withdrawn job ends its wait as a refusal and takes the request off the node's page. Any other answer keeps it waiting; nothing on that route can approve a job. These are a separate route family from /api/v1/management/*, which runs the other way — the plane calling in to a node's web tier — and stays status-only.

/artifact is the one response that is bytes rather than a JSON envelope, and it is described under Machines with no site. Its body fetches are metered on their own bucket (api_agent_artifact_rate_limit_requests, per node per window) and each is recorded, because they are the expensive ones; the small manifest fetches ride the general agent-channel limit.

The general limit (api_agent_rate_limit_requests per window) is counted in two buckets, one request-log row per request written at shutdown with the outcome. A request the node's signature proves (a result, a manifest fetch, a leave) counts toward that node's own bucket (api_agent_node, keyed node:ID); one that proves no node (a join, a missing or wrong signature, an unknown path) counts toward its address's (api_agent). A multi-tenant host's sites and a fleet behind one NAT share an address, so an address over its limit refuses only what proves no node: a neighbour whose signature verifies still gets in. A refused request is not recorded, so a bucket drains while it is over. The one request it does not count is a claim that succeeded: a fleet polls on a seconds cadence, which is tens of thousands of requests a day against a few hundred of everything else, and the rate-limit check counts rows with a query. A claim that fails does count, since an unsigned or mis-signed flood looks exactly like that. AgentChannelEndpoint::meterOutcome() is the rule.

Cloud account tabs

The dashboard at /admin/server_manager has two tabs, Main (our main cloud accounts) and Test (the disposable test account). Hosts, machines, the find box, Agents asking to join, Cloud provisions and Recent Jobs show only the selected tab's boxes. ?account=main|test selects a tab and a cookie remembers it. The other tab shows its node count and a dot when it holds an open incident that needs a person or a pending join.

An account is recorded on a host (mgh_cloud_account) or a node (mgn_cloud_account) as main or test. A node placed on a live host takes its host's account; any other node uses its own; an empty value reads as main, so an unmarked box is never hidden from the main tab. CloudAccounts holds the labels and the rules.

Main or test is the flag of the cloud account of ours the box is in (see Cloud accounts): a server created in, or adopted from, an account flagged test is test, any other account of ours main (CloudAccounts::for_provision()). A box made by hand is marked when it is first saved with the default account's flag (CloudAccounts::plane_account()), main when there is no default. Add Host and Connect Site ask for the account, with Main preselected. Migration sm_016_cloud_account_backfill marks the unmarked boxes a test-account plane created since 2026-10-07, the day its account became the test account; nothing is changed on any other plane.

A join belongs to the account of the box whose address it comes from, or of its provision. A join from an address no box or provision has appears on both tabs. A provision with no node yet belongs to the account it is in. The notices at the top of the page and the Incidents inbox are not filtered by tab.

Join approval

An agent joins a plane by asking: it sends its name, public key and address, and a join request waits on the dashboard's Agents asking to join panel. A person approves it there after checking the key against what the machine printed, or a join from a machine this plane provisioned is approved automatically.

Automatic approval (JoinAutoApproval, run each minute by the Auto-Approve Provisioned Joins scheduled task) approves a pending request only when all of these hold:

1. its source address (IPv4 or IPv6) belongs to a provision that is installing or done;
2. its claimed name is exactly that provision's site name (the site agent) or <site name>-host (the host agent);
3. its key equals the one the provision's install showed for that agent — recorded when the install job finishes (cvp_expected_site_key, cvp_expected_host_key), read from the host agent's Key fingerprint: line and the site agent's SITE_AGENT_KEY= line in the install output — and not yet used;
4. the cloud provider reports the instance running at that address;
5. the site node has no agent yet, or no host agent is paired on the machine.

The panel shows, for a request that matches a provision, why it is still there ("no key was recorded from this provision's install", "the key does not match the one the install showed", and so on); a key that differs from the install's is flagged in red. Joins from machines this plane did not provision are always approved by a person. A person approving a join from a provisioned machine approves it as that machine's site node, its host's node, or a site in a container on it, and only once the provider reports the instance running at the join's address. Site copies approve their own agent. Setting server_manager_auto_approve_joins switches the automatic approval off.

Machines with no site

Some machines a management node manages host no Joinery site at all — a mail relay, a Docker host. They run the same agent in a machine posture: no site root, no local database, no admin page, and no platform release ever delivered to them. Two things follow, and both are served by the same endpoint.

Keeping the agent current

A machine with a site tree finds its next binary in public_html/agent_dist, put there by its own upgrade, and updates without asking anyone. A machine with no site has no such directory, so it fetches from its management node instead: a signed request to /api/v1/agent/artifact for the dist manifest, and — when the version differs from what it is running — for the binary for its architecture.

Nothing about verification moves. The agent decompresses, checks the sha256, and verifies an Ed25519 signature against the public key compiled into its own binary at build time, exactly as it does for a locally delivered artifact. The management node does not hold the release key and cannot sign an agent, so a plane serving a hostile binary produces a refusal and a recorded rejection. The endpoint is a delivery route for bytes that were always verified on arrival.

The request names a kind from a closed set and, for a binary, an architecture matched against a pattern. It never names a file: the plane resolves what to send out of its own manifest, so nothing a node sends is read as a path.

The support bundle

A script-invoking primitive verifies its script against the signed release manifest before running it as root. On a site that manifest is RELEASE_MANIFEST and RELEASE_MANIFEST.sig at the site root, beside public_html; install.sh places it there when it lays down a fresh site (bare metal and the Docker image alike, from whichever archive it actually applied) and upgrade.php replaces it on every upgrade. A site tree without it refuses every script primitive, apply_update included. On a machine with no site there is no release manifest, so there is nothing to verify against and no script primitive can run at all — which would leave the machine's whole vocabulary in embedded Go.

Only the site that built the shipped agent signs. The agent verifies against the key compiled into its binary, so the key that built the bundle in agent_dist/ is the only one whose signatures any node will accept; agent_dist/manifest.json records it as signing_public_key. TreeManifestPublisher::authority() compares that key with the site's own config/agent_signing_key and the publish signs only when they are the same (a bundle that predates the record counts as built here when the agent source is on the box). A site that received its agent from upstream — getjoinery, republishing for the beta testers — holds a different key, and a manifest it signed is one its own agent and every node it serves would refuse. Such a site carries forward the manifest it received: into the core archive from its own site root, and for each theme and plugin from the directory that ships, leaving its live tree exactly as upstream delivered it. A site that republishes (it may not mint a release number) also builds each archive from exactly the files that manifest lists, each checked against its hash (TreeManifestPublisher::republish_artifact(); see docs/deploy_and_upgrade.md), and rebuilds nothing — the agent, the support bundle and the other binaries ride along as received. A received manifest that turns out to be signed with the site's own key, or that does not verify against the bundle's key, aborts the publish and says so, rather than republishing a tree nothing can run.

When a node cannot verify its own scripts

Verification fails in two ways, and they are opposite problems.

A file does not match its signed hash, or is not listed at all. The manifest is good and is doing its job: the file on disk is not the file that was published. That script refuses, and nothing repairs it automatically — find out why the file differs.

The manifest itself cannot be used — absent, unparseable, or signed by a key this agent does not carry. Nothing is known about any file, so every script primitive refuses at once, apply_update among them. A node in this state cannot be repaired by sending it work: the upgrade that would deliver a good manifest is refused by the check that is failing, and no non-script primitive writes a file.

The agent recovers from the second on its own. Every ten minutes it asks whether the core manifest is usable; when it is not, it reads the release its site runs from public_html/VERSION, asks the plane for that release's signed manifest over the artifact endpoint (release_manifest, naming an artifact and a version — never a path), verifies it against the key compiled into its own binary, and writes the pair to the site root. A manifest that does not verify is refused and not asked about again until what is on offer changes. The plane holds no release key and cannot sign one, so serving these bytes gives it no new authority — the same footing on which it already serves agent binaries. ReleaseManifestSource reads the pair straight out of the published archive, so any version still in static_files can be served; one pruned by retention cannot, and the agent says so rather than retrying silently.

Recovery is driven by ArtifactManifests.Usable(), which reports only on loading the manifest. A hash mismatch is raised by Verify() and never reaches it, so recovery is structurally incapable of firing on a modified file — the case where replacing the manifest would destroy the evidence.

The manifest is read back into the agent under an 8 MiB cap of its own; the 64 KiB cap that bounds every ordinary plane answer cannot carry one. The claim is the other answer with its own cap: the agent reports the most it reads (claim_bytes, 1 MiB from agent 1.45.0), because a long backup chain's jobs outgrow 64 KiB, and the plane holds each job it hands out to that (AgentChannelEndpoint::claim_body_limit, dispatch_refusal); a claim that reports none is held to 64 KiB. Each file is landed through a temporary file created with O_EXCL in the site root, with mode and ownership set on the descriptor rather than by path — the site tree is owned by the web user, so a root process writing there by name is a path a web-layer compromise can redirect.

A verifier is cached only while the manifest on disk is still the one it was parsed from, keyed on size, mtime, mode and inode. One agent build spans many core releases, so a parse held for the life of the process would have a routine upgrade reading as every file having been modified since release, and a manifest broken under a running agent staying invisible while the node reported itself healthy.

Scope is the core release. A plugin or theme carries its own manifest, and a node whose plugin manifest is bad still has a working apply_update, so an ordinary upgrade repairs it.

The state is visible without opening a job. A node reports script_trust on every poll, and a refusal that carries either wording is classified and stamped on the node as it arrives (mgn_script_trust, with when it was first seen, the reason verbatim, and the job type that refused). Such a node raises a critical incident (plane:unmanageable); its failing backups are a symptom, not a second problem. A node's own report wins over a stale refusal in both directions for the manifest. The report looks at the five deployment files and no other, so a refusal of any other script stays flagged until the job type that refused completes; an absent report — an older agent, a machine with no site whose bundle has not landed yet — is left alone and never reads as healthy. A machine with no site reports on the support bundle, the tree it actually runs scripts from, so it sets and clears the state the way a site node does.

The deployment files an upgrade replaces first. utils/upgrade.php copies five of its own files (utils/upgrade.php, utils/update_database.php, includes/DatabaseUpdater.php, includes/DeploymentHelper.php, includes/PackageSignature.php) over the live ones before anything else is checked, so a run that then stops short of a deploy leaves the old release with new deployment files in it. They no longer match the installed manifest, the agent refuses upgrade.php as modified, and every later apply_update is refused the same way. Three things keep a node out of that state. The upgrade itself puts the replaced files back, and empties staging, when it ends without the live VERSION having changed (DeploymentHelper::restoreSelfUpdateFiles). The agent, which lives outside the upgrade's process group, keeps a copy of each of the five that matches the installed manifest before every apply_update (/etc/joinery-agent/release-files/, tagged with the release) and settles afterwards, and on its manifest-check clock, whatever happened to the run: a file that no longer matches while the VERSION is unchanged is written back from the kept copy, never over a running upgrade (the upgrade lock is checked) and only with bytes whose sha256 is the one the key-verified manifest lists for that path. And the restore_release_file word does the same on demand for a node already in the state: its one parameter names one of the five files, the bytes come from the kept copy or, failing that, from this management node (artifact kind release_file, read out of the published core archive of the installed version by ReleaseManifestSource::read_file), they are checked against the manifest before a byte is written, every directory is opened relative to the one before it without following links, the file's mode and owner are kept, and the file it replaces is saved under release-files/replaced/. It refuses when the file already matches, when the manifest itself is unusable, and when no source has a verifying copy. The node asks this management node for the file by the sha256 its manifest lists (the release_file request's sha256; ReleaseManifestSource::read_file_by_hash searches every published core archive, the node's VERSION only a hint of where to look first), because the VERSION and the manifest can disagree: a deploy that fails its tier and rolls back puts the old code and VERSION back under the new release's manifest, and a file served for the VERSION's release is then exactly the one the node refuses. repair_release_files is the same for every differing file in one parameterless word, and the node runs it itself: recipe release_files checks the five files every ten minutes (upgrade_preflight's manifest and files facts; an unusable manifest is unknown, the healer's case, never repaired) and repairs after two failing checks, so a node in this state is upgradeable again within about twenty minutes with no one's hand on it.

The poll names the files. A node whose manifest is good but whose five deployment files differ reports script_trust as untrusted_file, and script_trust_files lists the ones that differ (mgn_script_trust_files). An empty list is the node saying it looked and every file matches, which is what clears the state once a file has been put back; an agent that does not report the list cannot clear it. While an upgrade holds the site's lock (uploads/.upgrade.lock) the poll gives neither, so the stored state stands: upgrade.php replaces itself minutes before the manifest changes, and a file differs on the way through. The agent asks the kernel's lock table (/proc/locks), never the lock itself, because upgrade.php refuses to start when it cannot take the lock at once; upgrade_preflight's lock check and the start-up recovery of the deployment files ask the same way. A refusal and the poll's list are two findings held side by side (the refusal in mgn_script_trust_reason and mgn_script_trust_job_type, the list in mgn_script_trust_files), the worse one setting the state, and each clears only itself: a job completing releases its refusal and leaves the files the poll still names, and the poll's empty list releases the files and a refusal of one of them, never a refusal of another script. The node page and the plane:unmanageable incident list the files, with Put them all back (repair_release_files, one job for every differing file) on a node whose agent reports it and Restore signed copy beside each on a node whose agent reports restore_release_file.

The publishing site's own changes are not tampering. The site that signs releases runs from the tree it signs, and its live manifest is rewritten only when it publishes, so a script committed or edited in between is refused until the next publish. When a file refusal, or a poll naming differing deployment files, arrives for this management node's own record, this site is a git checkout that re-signs its own tree on publish (TreeManifestPublisher::signsItsOwnTree()), and the refused file is either identical to its last commit or one the web server cannot write, nor any folder from it up to the site root (NodeMonitorHealth::beyond_this_process()), the state is unpublished_file instead (NodeMonitorHealth::changed_here_after_publish(), and file_changed_here_after_publish() for each file the poll names, every one of which must pass). Agent results are handled by the web server, so the second test asks whether the web user could have made the change, which is the compromise the file check exists to catch; the agent still refuses to run the file. It shows on the node page as a warning and raises no incident, and clears the same way after the publish: the job type completing, the poll's empty list. Any other node, a tree the web user can write (every node a release installs), a process that can write the file, or a site that cannot sign its own tree keeps untrusted_file.

The support bundle closes that. A publish builds public_html/agent_dist/support_bundle.tar.gz: a small tree carrying the scripts those machines' primitives invoke, at site-root-relative paths, with its own RELEASE_MANIFEST and .sig signed by the release key. A siteless machine fetches it over the same artifact endpoint, verifies the signature against its baked-in key, checks every listed file's hash and that the tree holds nothing the manifest does not list, then unpacks it root-owned to /opt/joinery-agent/tree. install.sh docker also installs the release's own bundle from agent_dist/ with joinery-agent bundle install FILE --statement=RELEASE_STATEMENT (agent 1.75.0), which holds the file to the same checks and the public log and then runs the converge a fetched bundle runs, so a Docker host that never joins a management node has its host timer too. On a joined machine the binary and the bundle are its management node's release: a bundle on offer replaces the one installed whenever their bytes differ, newer or older, as the self-update replaces the binary. Script primitives resolve against that tree when there is no site root; a machine with a site root uses the site root, and a machine with neither refuses as it always has.

Live proofs of agent words. A changed agent word runs once on a real machine before the release ships it (specs/fleet_security_patching.md WP7); the publish page warns about one that has not (AgentLiveProofs, agent_live_proofs.json). The live-tier test agent_live_proofs_live runs them (AgentLiveProofRun). It builds the agent from the source being prepared and the support bundle from this tree with the release's own publishers, in a scratch site whose only release key is one made for the run, so the binary differs from a release's only in the key baked into it and nothing the fleet trusts signs anything. It makes a throwaway server in the test cloud account (test_server.php), installs the agent (install_agent.sh --siteless) and the bundle over SSH, joins it to this management node, approves the join as the fingerprint the machine printed, only when it came from the server's own address (the address this plane saw it arrive from, never what the machine reports) under the run's label, and marks the node a test build (mgn_agent_test_build). This plane offers such a node none of its own agents or bundles (AgentChannelEndpoint::OWN_BUILD_KINDS, answered as a plane with none has), which it would refuse, each refusal an incident. Each word is queued as a job and judged (completed with an answer to read; for install_security_updates, an outcome that installed or found nothing to install), and a passing word is recorded with the build's agent version. The node, its join requests, the server, its DNS and the run's key are then removed, whatever happened; each run builds in a fresh scratch site, so a killed run's key is never reused. Only the words in AgentLiveProofRun::RECIPES can be proven on a bare machine. The words a site runs (SITE_RECIPES: upgrade_preflight, apply_update, site_log) are proven by the live test agent_live_proofs_site_live: the agent is built trusting the run's key and the real release keys; a candidate core archive is built from this tree as a publish stages it (build_candidate: publish's exclusions, which a safe test pins to publish_upgrade.php's own, the licenses, the config template, install_tools and sysadmin_tools, the run's agent in agent_dist, the next version, no release statement), signed with the run's key and checked against it; the newest published release is installed on a throwaway server as the provisioning flow installs a bare-metal site; the run's agent replaces the release's, its dist pinned to its own so it never updates back; and upgrade_preflight, apply_update onto the candidate (served from the server to itself, the run's key added to that site's trusted keys and the log requirement lifted there alone) and upgrade_preflight again run through it, judged by StagedRolloutRunner's own preflight_verdict and gate, then site_log reads the upgraded site's access log. The installed upgrade.php self-updates to the candidate's and re-runs it, as every node's upgrade does. A changed word in neither list (publish_upgrade, which only the publishing box runs) is proven by hand with record_live_proof.php.

The bundle carries the host installers and nothing that reads a site, so a site question never goes to a machine in machine posture: the status processor asks a node for its recovery key only where ManagedNode::hosts_site() says there is a site to ask about, the same test the recovery-key page applies before calling a node not applicable. A site primitive that reaches such a machine anyway is refused as a posture, in words the trust classifier leaves alone ("this machine has no site, and its support bundle does not carry ..."), never as a file that fails its release.

The bundle's contents are a deliberate list in SupportBundlePublisher, not a directory sweep — every entry is a script some primitive names, and adding one is a visible decision. The Docker host is its consumer: decommission_site runs the bundled remove_account.sh, host_report reads the machine with host_report.sh, and host_converge runs the bundled host runner (_plugin_installers_start.sh --machine) over the two host installers, host_housekeeping.sh and install_host_converger.sh, with _tree_trust.sh beside them and the Cloudflare range list at its site-root-relative path. That is how a machine with no site converges its own fail2ban and installs its own host timer; the site installers (agent, parser jail, vhost, permissions, secrets) are deliberately not in the list, since a site installer on a machine with no site has nothing to install. A script that sources another needs that sibling in the list at the same relative path, and so does every file the functions it calls there use beside them: setup_ssl.sh sources install.sh, whose certificate retry is armed by arm_ssl_retry.sh, so both ship. The safe test agent_machine_scripts_bundled follows each bundled script's calls into install.sh and fails on a file the bundle leaves out. Binaries the scripts invoke ship under bin/ with one file per architecture (bin/<tool>-linux-amd64, bin/<tool>-linux-arm64) and the script selecting on uname -m: one bundle carries both, so no machine can end up holding one built for the wrong architecture.

The list runs from the publishing box, so SupportBundlePublisher.php is a held path (ReleaseCommit::heldPath(), with publish_upgrade.php, ReleaseCommit.php and every other *Publisher class): while it has an uncommitted change, the publish refuses. The scripts themselves come from the commit. A bundle that cannot be rebuilt (a listed script missing from the commit) carries the previous one forward with a warning; a release that would ship no bundle at all is refused, a republish too.

Its version is the hash of its own manifest body, so a publish that changes no bundled script leaves the bundle byte-identical and no machine downloads anything. The plane also advertises the tarball's sha256, which an agent uses only to skip a transfer it already has — what makes the tree runnable is the signature inside it.

Each machine reports the bundle version it holds on every claim, stored as mgn_agent_bundle_version. On a machine nobody logs into, that column is the evidence the bundle arrived.

Enrolling and switching one on

There is no admin page on these machines, so the ceremony is the same one reached from the command line:

joinery-agent join --management-node=https://plane.example.com [--name=NAME] [--no-wait]
joinery-agent status
joinery-agent enable | disable
joinery-agent leave

join generates the keypair, sends only the public half, and prints the fingerprint to compare against the pending request on the plane — the same comparison, and the same approval, as a node enrolling from its own admin page. --name is what the plane's pending list shows for this machine (default: the hostname, which on a fresh VPS is usually localhost). Without --no-wait the command waits up to five minutes for the approval; with it, the command returns once the ask is lodged. Either way the running agent finishes the join itself: it watches the staged keypair, and when the plane approves — minutes or hours later — it stores the credential and starts polling without a restart or anyone at the terminal. install.sh docker --management-node=URL --node-name=NAME is how a plane-built host asks. leave deletes the credential; the running agent sees it gone within seconds and restarts without it, so a join after a leave is finished the same way. The run switch is the marker file /etc/joinery-agent/enabled, which enable and disable write directly: on a machine with no settings table, the marker is the switch rather than a projection of one.

A host agent is born at the Docker install and nowhere else. install.sh docker places it (install_agent.sh --siteless --dist-dir=<tree>/public_html/agent_dist --enable, the one caller of --siteless in the tree, pinned by installer_contract_test) after the host's housekeeping, and with --management-node=URL [--node-name=NAME] asks to join in the same run; approval on the plane links the host record to the new node. Nothing later installs one: no upgrade, container start, plugin installer or plane job reaches a host that has no agent, and once installed the agent keeps itself current from the plane. A host built before the Docker install placed agents gets one the same way, by running install.sh docker --management-node=URL --node-name=NAME once from a current release tree copied onto it; that run's already-installed branch does the housekeeping and the agent install and nothing else. --siteless is explicit and never inferred — a missing site config keeps meaning "not my machine, exit 0" for everything else.

The Docker host as a node

A shared Docker host is a plain ManagedNode in machine posture — paired, addressed and versioned like any node, with no web root and no container name. install.sh docker installs the host's own siteless agent as part of the install and, given --management-node=URL (and --node-name=NAME for the pending list), lodges its join; the machine that runs our Docker is managed by its own agent, which is the only path to certificate renewal or site removal once SSH is gone. The placement record (mgh_managed_hosts) stays what it is: which containers live where. One nullable link joins the two worlds: mgh_mgn_managed_node_id on the host record names the host's own paired node. Approving the host agent's join sets it, and so does each host report the machine sends (ManagedHost::link_host_node, from this plane's own records: the placement record the machine is itself placed in, when it has no host node; otherwise, for a machine that is no record's agent yet, the first record for its address, unless that one has a live host node already, which is never re-pointed), and the host's edit page sets it by hand. That is the routing chain for host-scope work — a container victim's mgn_mgh_managed_host_id finds the host record, the host record names the host's node, and the job is addressed there. Sibling containers on a host are found by mgn_mgh_managed_host_id and nothing else; ManagedHost::ensure_for_node() mints or links the placement record the moment a node needs a container port, so the FK is never absent where it matters. The writers that see every kind of node — the add/edit form and join approval — go through ManagedHost::place_node(), which mints a record only for a node that names a container and otherwise just links a record that already exists at the address (a host's own agent landing on the record its containers point at). A bare machine gets no placement record: it is only a node. A site agent running inside a container says so, on its join and on every claim (container, agent 1.70.0 and later); the plane then names the container for the site folder in the node's web root and places the node on its Docker host's record (ManagedNode::adopt_reported_container()), so a site that joins a management node is grouped under its host the same as one that node created. A node already named and placed is left alone; one named before placement existed and never placed is placed then, on the record for its own address. A machine with no site never says it.

A host record can be deleted (soft) from its edit page — last, deliberately: the delete refuses while any container site still names it as placement, and while its own agent node record is live.

Removing a container site (decommission)

Removing one site from a shared Docker host is host-scope work — the container, its own {site}_<name> volumes (the fifteen install.sh creates, by exact name, so a sibling site named {site}_<word> keeps its own), the reverse-proxy vhost and the web root all live outside the container — so it runs as the decommission_site destructive primitive on the host's own agent. The wire carries exactly one parameter, the site's name (^[a-z0-9_-]{1,50}$); every path is composed on the host from compiled-in patterns, and the plane cannot express one.

The site being destroyed approves its own removal. The host agent locates the victim through two host-owned files — the vhost names the container's published web port, and the config volume's host-side path (/var/lib/docker/volumes/{site}_config/_data/) carries its database credential — then connects over the container's published loopback database port (web port + 1000) and stages an approval into the victim's settings table, sealed to the victim's proven backup recovery key. The victim's own Backups page renders it (ApprovalChallengePanel, decommission scope: this site will be destroyed, permanently), with the site's own record of its last completed offsite upload as the load-bearing fact. The ceremony has its own settings rows, HKDF context and plaintext tag (decommission_approval_*, joinery-decommission-approval:), so a decommission answer and a restore answer can never satisfy each other. The plane is not in the path: the primitive declares no parameter an answer could travel through. The victim's credential is held in memory for the life of the connection, never at rest; everything read from the victim is treated as untrusted display data.

Only after the answer verifies does the host run the bundled, self-verifying remove_account.sh, whose output carries its own verdict (DECOMMISSION_VERIFIED / DECOMMISSION_FAILED_VERIFY). Before it deletes the vhost, the script reads the certificates the vhost names (each /etc/letsencrypt/live/<name>/ and /etc/ssl/joinery/<domain>); once the vhost, the site's certbot-made <site>-le-ssl.conf and the vhost's backups are gone, it removes each through its sibling remove_site_certificate.sh, which keeps any certificate anything else on the machine still uses or serves, so certbot stops renewing a domain that has left the host. A certificate it could not remove is a warning in the output, not a failed verdict. It removes only a container install.sh made: one running the site's own image (joinery-<site>, any tag) with every mount one of the site's own named volumes. Anything else (another image, a bind mount, another site's volume) is refused before anything is removed; a container made by hand is removed by hand. The script touches nothing of another site on the machine: it checks the name's shape, stops or removes a container only of exactly that name (Docker would otherwise fall back to a container ID starting with it), removes the site's volumes by exact name (plus <site>_postgres_pg<major>, the old database a rebase keeps, its pre-rebase images and /root/rebase/<site>), and treats <site>_test as a bare-metal site's test twin and <site>-le-ssl.conf as its certbot vhost only when no site of that name exists; a <site>_test with a vhost of its own is the twin only when that vhost serves test.<the site's domain>. On bare metal it drops the site's databases with the password from the site's own config (never a prompt), holds the host converger's lock for the whole removal, removes the host converger when it ran for the site, and checks the databases gone before it says DECOMMISSION_VERIFIED. Behind a container's proxy (its container found, or its vhost a ProxyPass to 127.0.0.1, as on a re-run after the container went) it drops no host database and takes no test twin. Docker installed but not answering, or not on the script's PATH while its socket or service is there, refuses before anything is removed, and stopping mid-run fails the verdict. Only plain files are cleared as marks, so a quiet bare-metal site's held/ folder (_site_state.sh) is left alone. build_decommission_node refuses a container that another live site row on the same host, or on no recorded host, names (compared without case or spaces), since a row's container name is editable. Dispatch marks the victim quiet (mgn_agent_quiet_time) so its agent dying with the container reads as a deliberate silence.

The old machine of a switch-over is removed on the host's proof instead. After a switch-over the domain points at the new server, so the old container's Backups page cannot be reached to approve anything. For a row in state retired, build_decommission_node sends decommission_moved_site (agent 1.55.0) to the host instead, with the same one parameter and the same script. The host writes a one-time token into the container's public_html/sm-ssl-probe.txt with docker cp, which runs nothing inside the container. It checks that the container serves the token on its own port. Then it fetches each name its own vhost proxies to the container (the ServerName and any ServerAlias) over https, with the certificate checked, no redirect followed and a fresh query string. The token coming back refuses: the domain still reaches this site. No answer, a certificate error or a 5xx refuses too, because unreachable is not moved. So does an answer without Joinery's X-Joinery-Version header, which serve.php sets on every answer it routes (the probe view's 404 included): a CDN edge's challenge page or its cached copy of a live site, or a parking page, is not another Joinery server. An answer below 500 with the header means another Joinery server holds the name; an edge passes the header through on a real answer, so a site that moved behind a CDN still proves. An alias that does not resolve at all passes; the ServerName must answer. The token file is emptied either way. A switch-over leaves the old container frozen and then held stopped on its host (hold_container), so it can serve no token. When the hold is recorded on the host (/etc/joinery/sites/{site}/held) and the engine says the container is not running, the host instead asks each name the same way and needs an answer below 500 with Joinery's header, and none of the name's addresses may be one of the host's own (compared as addresses, IPv4 and IPv6): nothing on the host but its own proxy can answer for those names, and that proxy answers 503 for a stopped container or a suspended site. The hold is no evidence the site moved, since this management node can hold any site container; it only says the container cannot answer. The address rule reads the host's interfaces, so on a host whose public address its provider NATs to it, the header and the proxy's 503 carry the proof alone. A copy onto the same host never passes it (its names resolve to the host), and its old container is removed with the site's own approval. That takes a host agent from 1.80.0 (JobCommandBuilder::MOVED_SITE_HELD_MIN_AGENT_VERSION), which build_decommission_node requires for every retired row. The proof runs on the host, never here, so this management node alone still cannot destroy a site: it only chooses the word. Because the host's proof passes for any other server answering (a parking page included), build_decommission_node also requires this management node's own proof that the domain reaches the site's new server, from the last ten minutes (MovedSiteCheck::new_server_refusal). That proof can refuse a removal but never allow one the host's proof refuses. The victim needs the routing-probe route (ProvisionPendingSsl::PROBE_MIN_CORE_VERSION) rather than the approval panel.

The page shows where the old machine's domain goes. Beside the site on the overview of a retired container row, MovedSiteCheck shows two answers together. The host's: the domain left this container, still reaches it, could not be told, or the container is gone. It is the same proof, run as moved_site_check (agent 1.56.0), which reports its verdict instead of refusing and removes nothing; process_moved_site_check folds it onto the row's mgn_moved_check_state, mgn_moved_check_detail and mgn_moved_check_time. This management node's: whether the domain reaches the new server (the row named by mgn_copy_of_node_id). A moved_site_reach job places a one-time token on the new server through its agent (ssl_probe_place); process_moved_site_reach fetches https://<domain>/sm-ssl-probe.txt (certificate checked, no redirect, fresh query string), records reached, elsewhere, unsure or failed on mgn_moved_reach_state, mgn_moved_reach_detail and mgn_moved_reach_time, and files moved_site_reach_clear to empty the file. The site reads Moved only when the domain left the container and the token came back; left-but-not-reaching reads amber with what the domain answered. The page shows the stored answer at once. When it is older than ten minutes, the page asks again through the moved_site_check API action and polls until the host answers; Check again asks at once. A host check refuses while a removal is open on the host, because both write the same probe file. A verified removal records the container gone (absent), which is never asked again. A DNS lookup cannot answer this: a proxied domain resolves to the proxy for the old server and the new one alike.

build_decommission_node routes or refuses, each refusal naming the operator's next step: a relay goes through the relay flow; a bare-metal node is a whole machine — delete the instance at the provider, then delete the node record; an unpaired host says "pair the host's agent"; a victim below the core release that carries the approval panel (DECOMMISSION_PANEL_MIN_CORE_VERSION) says upgrade it first; a victim with pending or running jobs says finish or cancel them. There is no unattended path to destroying a site — a victim that cannot render its own approval does not take this path.

Incidents

An incident is one condition on one node that needs a person, from the moment it starts until it is over. Every incident carries two answers, kept apart:

• Is the condition still there? Its source says: happening now or cleared (inc_status, open or closed). Nothing a person does changes it. A source is the node's agent (a recipe that gave up opens a case, see The case under the agent channel, and the node closes it when its check passes) or this management node's own detectors (below).
• What is being done about it? A person says: new, snoozed until a time, resolved or ignored (inc_triage). A new incident starts new, and Reopen sets it back to new. A snooze that has ended reads as new again (derived on read: a page view never writes). Ignore covers that incident only; the next time the condition happens is a new incident.

An incident needs you when it is new, or snoozed and its time has come (IncidentRecord::NEEDS_YOU_SQL, one definition). Needs you holds only what is still happening: an incident that clears while it is new or snoozed is resolved on the next pass, by no person, with It cleared before anyone looked, so nothing needs doing on its timeline (IncidentReconciler::settle_cleared(), which covers an agent case the node closed too). The open signal already told the superadmins; if the condition comes back within the hour the incident reopens as new. Severity is critical (visitors or data at risk now) or warning; an agent case is a warning.

Where a person sees it. One line at the top of every admin page (IncidentNotice, registered as fleet_incidents; it is the only fleet line in the header): how many incidents need you and how many are still happening, beside the number of problem reports from sites that need you (the bug_reports plugin's own notice yields to it for superadmins). Red when a critical one that needs you is still happening, amber when any incident needs you, blue when only problem reports wait, absent otherwise; it names no nodes, so it stays one line. The Incidents menu entry carries the same count (AdminMenuCounts), and the node page has an Incidents card and the same count for that node.

The pages. The Incidents page has a view per triage (Needs you, the default, then New, Snoozed, Waiting for proof, Resolved, Ignored, All, each with its count; every incident is in exactly one of New, Snoozed, Waiting for proof, Resolved and Ignored), filters by node, type and still happening, and a dropdown to set several selected incidents at once. An incident's page shows its state, its timeline oldest first, its evidence (an agent case's reason, the failing checks since, what the node tried and its host at the time), the triage buttons, a snooze (4 hours, 1 day, 3 days, 1 week) and a note box.

The record. Incidents live in inc_incident_records (IncidentRecord): the source's columns, plus inc_title (the plane's own words, IncidentTitles), inc_severity, inc_triage, inc_triage_time, inc_triage_usr_user_id, inc_snooze_until and inc_detail (what a source other than the agent saw). What happened is in ine_incident_events (IncidentEvent): opened, cleared, reopened (from the source), triage and note (from a person, with who; a triage with no person is the reconciler's: unproven in its data sends an unproven fix back to new, settled resolves an incident that cleared before anyone looked), and analysis. An event records a change, never a tick: an agent case's repeated failing checks stay folded on the incident.

This management node's own sources. Besides the agent's cases, conditions this management node detects itself become incidents. Each is an IncidentSource registered from the plugin bootstrap (IncidentSources::register), named plane:<name>, that answers from what is already stored on the node's row and never probes. The sources:

Telling people. An incident that opens or reopens, or a fix sent back to new for want of proof, dispatches one signal, never one per tick: incident.opened for a warning, incident.opened_critical for a critical one, addressed to every superadmin. Notify puts it on each one's bell; email is on by default for critical and off for warning, and each superadmin can change either on the notification preferences page. An ignored incident that reopens tells nobody.

A node's alerts. Whether a node's incidents tell anyone is the node's Alerts setting in Connection Settings (mgn_alerts, ManagedNode::sends_alerts()): Automatic (empty) is off for a box hosted in our test cloud account and on for every other node; On and Off say so for that node alone. A node whose alerts are off still opens, refreshes and clears its incidents, which wait in the inbox and count in the header line as any other; it dispatches no signal, so nobody gets a bell, an email or a push. Test cloud boxes are made and removed by the dozen, and nearly every one starts with a condition nobody means to fix (no verified recovery key), so they are quiet unless someone turns a box's alerts on to watch it.

Analysis. Where the Joinery AI plugin is active, an incident's page has Analyze: a model reads the incident (title, timeline, the source's evidence) and the node's facts (release, agent version and recipes, uptime, host report, recent jobs) and says what it thinks is wrong. Pressing it records an analysis event marked running and starts a CLI worker (cli/analyze_incident.php, IncidentAnalyst::run); the page shows it running and refreshes until it is done. The model may ask the node for evidence with one tool, observe, from a fixed menu of the agent's read-only words (host_report, disk_usage, unit_journal, site_log, check_status), at most three requests within ten minutes. Each is an ordinary job built by JobCommandBuilder with its own checks (the owner's log-access switch, the compiled unit and log lists), redacted on the node. Anything else is refused and recorded. It answers with conclude: the likely cause in plain words, how sure it is (low, medium, high), the evidence it used, and one next step from a closed list (restart a service, run a backup, read a log, run a status check, a person needs to look), stored on the event and shown escaped. It never acts. The model comes from joinery_ai's resolver (purpose incident_analysis, the site's selection policy, local first, tools required, the person's private-content consent as the trust floor), under joinery_ai's monthly token ceiling, and its cost-bearing tokens count toward that ceiling (IncidentAnalyst::usage_since, registered with CostGuard). Node text reaches the model inside joinery_ai's untrusted envelope. An analysis whose worker died stops counting as running after fifteen minutes.

Writes. Every triage change and note goes through IncidentTriage: the pages post to themselves with the admin CSRF token, at 8, and the incident_triage API action (/api/v1, 8, not callable by the AI agent) takes the same do values (resolved, ignored, new, snooze_4, snooze_24, snooze_72, snooze_168) and notes. Setting the triage an incident already has records nothing. Everything a node or a note said is escaped where it is shown (IncidentViews), and nothing it said becomes a link.

Each machine's transfer

A cloud provider adds every machine's outbound allowance to one account pool, so the pool alone hides one machine running far past its own share. Watch Machine Transfer (daily, MachineTransferWatch) reads every account this management node holds a credential for (HeldCloudAccounts::drivers(): each of ours, and each connected customer account through ProvisionCustomerCloud::account_driver()), lists its machines (CloudMachineTransfer::listInstances()), and matches them to nodes by address: the node's host (resolved when it is a name), its server's host where it is a site on a shared one, and the addresses its agent reported at its newest approved join (a host name behind a proxy resolves to the proxy). A node is on one machine: where its addresses match more than one (a site that moved keeps the old machine's address on an old join), a match by host beats a match by a join's report, and of equals the newest machine wins. A connected account whose token cannot be refreshed is reported in the run's message and never marked for re-granting by this read. For each machine a node runs on it reads the provider's figure (getInstanceTransfer(); on Linode GET linode/instances/{id}/transfer, which needs only linodes:read_only) into one MachineTransfer row (mtr_machine_transfers): this month's bytes, the allowance (prorated for a machine created during the month), the overage, and the read before, whose difference is a day's rate. Every node on the machine points at the row (mgn_mtr_machine_transfer_id); a failed listing unlinks nothing. A machine no node runs on is not read. On the first two days of a month, a figure no lower than last month's last reading is taken as the provider's month not yet started again and is not recorded.

The provider's figure counts about the payload, and Linode leaves out traffic to another machine in the same data center over IPv6, in both directions.

One node speaks for each machine (mtr_mgn_managed_node_id): the server's own node where it has one, else the lowest-numbered site. plane:machine_transfer opens on that node only, naming the other nodes on the machine, so a server of twelve sites is one incident. Its conditions (MachineTransferWatch::conditions()), worst first:

Reverse DNS

A node whose server this management node created has a Reverse DNS box on its Overview tab: what each of its addresses answers now, and Set Reverse DNS, which names them through the cloud account the server is in (NodeReverseDns::set(), action set_reverse_dns). A provider takes only a name that points at the address, so the hostname's A record must give the IPv4 and its AAAA the IPv6; this is checked first, and the refusal names the record to create. A server with an IPv6 chooses what to name: both addresses (the IPv6 left as it is when the name has no AAAA giving it), the IPv4 alone, or the IPv6 alone (ADDRESSES_BOTH, ADDRESSES_IPV4, ADDRESSES_IPV6). The one address named alone must take. After an IP swap, the IPv6 alone is how the site's name goes on the copy's IPv6 while the moved IPv4 keeps the name it brought.

Adopting a cloud server

Reverse DNS and an IP-swap switch-over reach a node's server only through its provision record (cvp_customer_cloud_provisions), and creating a server is what writes one. A node that joined from another management node runs on a server this one did not create, so it has none. Adopt cloud server on its Overview tab (CloudServerAdoption, action adopt_cloud_server) records that server.

The panel appears in place of Reverse DNS on a node with no provision holding an instance, unless it is a container site: a shared server belongs to the server's own node. It names the machine the daily transfer watch matched, when it has one. On confirm, adoption asks the providers again:

• It lists the servers on every account this management node holds (HeldCloudAccounts::drivers(): each of ours and each connected account).
• It matches the node by address, by the same rule as Each machine's transfer. A match by the node's own host beats one by its join's report.
• Exactly one server at the strongest match is adopted. None, or more than one (each is named), adopts nothing. So does a server different from the one the panel showed, or a server another node's provision already records.

• the instance id, its IPv4 and IPv6, and its region;
• hosting mode operator with the account of ours it was found in, or customer with the connected account;
• that account's provider (cvp_provider), whose driver the IP swap and reverse DNS then use. An account whose provider has no driver here adopts nothing;
• the link to the node.

A later join from the adopted server's address is checked with the provider like any join from a server this management node created.

Fleet security

/admin/server_manager/fleet_security shows every machine and site container and whether it is patched (FleetSecurityView), red rows first:

• A machine's row comes from its host report: its operating system, the security fixes its hourly check found waiting (security_pending), what still runs replaced code (services_needing_restart), when automatic updates last ran, and a reboot owed.
• A site container's row comes from its own status check (its PHP, and whether its php-fpm runs the binary installed) and from its host's report.
• Every node's own status check also names the packages a package run left half done (dpkg_unfinished), which nothing else reports, since a site container has no hourly check of its own. Its operating system is its host's base image, so the fixes it waits for are the ones the host's base check found (site_refresh).
• Red: old code is serving (a php-fpm on a replaced binary, or PHP-FPM, Apache or PostgreSQL among what needrestart names, as the plane:old_code_serving incident counts it), or a critical fix waits (PHP, Apache, OpenSSL, PostgreSQL), or one was left half done.
• Amber: other fixes wait for the next run, a kernel fix waits for the night and then a person's reboot, a reboot has been owed over a day, no check has passed in two hours, any other service runs replaced code until it restarts, or a package run was left half done (Install security updates now finishes it). Packages half done under a run that still holds its lock are its work in hand, for two hours; past that the run is shown as perhaps stuck.
• Grey: nothing has been reported to judge by; a node with no agent (the relay, the DNS servers) says it has none.

• Each machine runs its own install_security_updates, one at a time.
• Then each container host whose nightly refresh is on replaces its site containers once, as its Refresh now (refresh_site_containers): a fresh base when a fix waits, then each site due, down about 12 s each. The step names the containers it covers, and waits for a host report whose run began after the host was asked and has ended, asking for a report every five minutes. A run already under way when asked is waited out, and the host asked again. A run that replaced what was due, or found nothing due, passes, noting a site passed over or held; a failed check for fixes, a base that did not build, and a run that stopped on holds halt it, as does a host report ten minutes after the asking that shows none under way and none recorded since (the run ended without recording what it did, or never began), and one not seen to end within its own bounds.
• A site container whose host does not replace its containers runs its own install_security_updates after its host. One whose host is replacing its containers after all is left out, or skipped while its install is open: its new container carries the fix.
• A node whose updater failed stops the run, naming it. One with nothing to install, or with another package run in the way, is recorded and passed.
• A node switched off, or whose agent lacks the word, is left out by name.
• Stop takes effect between nodes.

Machines

/admin/server_manager/machines lists every machine we pay for (MachineInventory), from three sources:

• Managed: each node that is not a site in a container, in whatever install state (a dormant copy and a retired source are machines we pay for, and the row names the state), with its operating system from its own host report. A node with no agent (the relay and the DNS servers, watched from here and never given one) reports nothing, so a note written on it (fhm_instance_ref node:N, from its row's Write down) says what it runs.
• Not managed: each server in our own cloud accounts this management node holds a key for that no node runs on (MachineInventory::lists_account(): not the test account, whose servers are thrown away nightly, and not a customer's Connect grant). A server is matched to a node by its addresses, as the transfer watch matches them. It shows its IPv4, or its IPv6 where it has none, and the release a note on it says it runs, or else the release of the image it was made from.
• Written down: each machine written down by hand (FleetHandMachine, fhm_fleet_hand_machines, at /admin/server_manager/machine_edit). That is a machine in an account no key here reads, or a note on a listed server that is not managed on purpose: what it is for, whose it is, what it serves, its release and a retire-by date. Saving it confirms it as true now, and Confirm does that alone.

• Red: an Ubuntu release past standard support (MachineInventory::UBUNTU_SUPPORT_ENDS, which holds the first day of the earliest end Canonical's pages give), or a retire-by date passed.
• Amber: a server no node manages and nothing written down explains; a node with no agent and nothing written saying what it runs; a machine written down and not confirmed for 90 days; a written-down machine that serves nothing written and has no retire-by date; a retired source, kept for the way back; or a node whose install failed.

Staged rollout

/admin/server_manager/rollouts applies the release this management node serves across chosen nodes, one node at a time, in the order given, and moves on only when a node's structured apply result (the apply in its apply_update job's result) says: the job completed, the deploy tier passed, the release's version is reported, and nothing rolled back. The first node that misses any of the four halts the rollout, which names the node and the reason; the nodes after it are never started. Put the node you would least mind breaking first.

A rollout is a tracked record (StagedRollout, table srl_staged_rollouts): its steps, each with its node, its stage, its machine, its apply job and a verdict (pending, running, passed, failed, skipped), survive a reload. A rollout runs in stages: a stage starts only when every node of the one before has passed, and within a stage the nodes go at once, one at a time on each machine (FleetBackupPolicy::machine_key: container sites on one Docker host are one machine), because an upgrade's syntax sweep, unpacking and tests load the machine it runs on. A rollout started on this page gives every node a stage of its own, so it goes one node at a time; Deploy to all managed nodes uses two (below). StagedRolloutRunner::advance() moves the current stage on each tick: it judges the steps under way, starts a waiting step on each machine that has none running, and moves to the next stage when every step has passed. It is called by the Advance Staged Rollouts scheduled task every tick and by the page on load. The first miss stops anything further from starting; the applies already under way are still judged, the page says the rollout is halting meanwhile, and it ends halted, naming the first node that missed. Stop ends it at once: an apply already running finishes on its node, and nothing further is queued. One rollout runs at a time. A node whose site has a copy being made, switching over, or switched and not yet kept waits, its step saying why (StagedRolloutRunner::copy_hold()): the copy must run its site's release, and a way back still open returns the old server to its data. A dormant or stopped copy holds nothing. A node is refused up front when it hosts no site, has been removed, or does not report apply_update (the standard needs a newer agent state); a node below the version floor is offered, since apply_update is what raises it. A node whose upgrade.php sends no structured result halts the rollout: an apply that cannot be judged is not a pass.

Readiness before each apply. When a node's agent reports upgrade_preflight, the rollout asks it whether an upgrade would stop before it started, and queues the node's apply only after a good answer. The word checks the signed manifest, the five deployment files, the VERSION, the upgrade lock, room to stage a release, and that the active theme is on disk, and answers pass or fail with a short reason for each. A failing check halts the rollout at that node with the check's reason and nothing changed on it; a check that does not finish in fifteen minutes halts it too. A node whose agent does not report the word is applied directly. A node that refuses its apply because its upgrade script no longer matches the signed release halts the rollout with a reason naming the file and saying another release will not repair it.

Pages to check. A rollout may name up to five pages (StagedRolloutRunner::MAX_PROBE_PAGES), one a line on the form, each a path with its viewer before it (admin /admin/admin_users; anonymous otherwise), kept on the rollout (srl_probe_pages). A node that cannot render a page (page_probe) is refused up front. Each node renders them as page_probe jobs before its apply is queued (probes_before) and again once the apply passes its gate (probes_after); a round that does not finish in fifteen minutes halts the rollout. StagedRolloutRunner::probe_verdict() halts at the node when a page got worse with the apply: it answers an error status (400 or more, or none) it did not answer before, raises more PHP errors than it did, or fails to load an asset it loaded; and when a page's check after the apply did not run. A page with no check before is held to being clean after.

Deploy to all managed nodes. The publish form's Deploy to all managed nodes checkbox starts a rollout of the new release as soon as the publish completes, so publishing and upgrading the fleet is one click. The choice is recorded on the publish job (deploy_all in its parameters, never sent to the node); the Advance Staged Rollouts task, and loading the publish job's page, start the rollout (StagedRolloutRunner::start_pending_deploy_all() / deploy_all_for_publish()). The rollout covers every node that hosts a site, except this management node itself, which is the release's source, in two stages (StagedRolloutRunner::deploy_all_plan()). The first is the sites marked Test releases here first on their node's settings form (mgn_rollout_first); with none marked, the first site by name, so a release never goes everywhere untried. The second is the rest, plain sites then management nodes, so a management node on a shared machine takes its turn last: one that breaks takes its own fleet's dashboard with it. A test site that fails stops the rollout before the rest are touched. The checkbox, the node's Updates tab and the publish job's page say which nodes are in each stage (deploy_all_description()). A disabled node, or one the rollout would refuse, is left out and named, both under the checkbox and on the job page. What became of the deploy is written back onto the publish job (deploy_all_outcome: started with the rollout's id, or refused with the reason), and the job page shows it. A publish that fails deploys nothing. If another rollout is running, the deploy waits for it to finish. A completed publish whose rollout has not started within 24 hours (DEPLOY_ALL_WINDOW_HOURS) is never started. A publish whose version is not the newest published release is refused, because it is not what a node's apply would pull.

Site copy

A node's Copy tab puts its whole site on another server and keeps it there, dormant and current (specs/site_copy.md): the database, the files, custom themes and plugins, the sealed secrets (which open there, because the copy takes the source's secret_box_key), the certificate and the DKIM keys. It is made from the backup chain this management node takes of the site, so every copy is also a real restore of those backups. The source does not change. The copy serves no visitors, sends nothing and runs no scheduled task (the quiet state, The quiet state in Deploy and Upgrade); switching over to it is a separate action.

What copies where. Any site copies to any of four kinds of place, and its address follows it by whichever way works there:

Alive or dead is the source's (SiteCopy scp_from, whatever the target). A running site (source) is frozen at the switch-over and copied one last time, so nothing written before the switch is lost. A site whose server is dead, or will not come back (backups, the tab's "From its backups"), is asked nothing and never frozen: what was written after its newest backup is lost, and the tab shows that backup's age.

Preflight (SiteCopyRunner::source_refusals(), read-only). A node can be copied when it is a site in the standard layout (/var/www/html/<site>/public_html) with an https site address, is a working site, runs release JobCommandBuilder::COPY_INSTALL_MIN_VERSION (0.8.497) or newer (the copy installs the source's own release, whose install.sh must take every option the copy's install passes: --outbound-set-by and an encrypted data root's options onto a new server, --release-tree-only onto a host; that release also carries the backup broker a copy run backs the source up through), is backed up by this management node (a site that backs up only to a bucket of its own cannot be copied), has a manager-profile chain of any age (its size sizes the new server), and has no copy already. A running one's agent must have been heard from within two hours (ALIVE_WITHIN_SECONDS), since it is frozen through it; it must hold a proven backup recovery key, and report backup_run, copy_export and site_census when its copy is made by export (anything but onto a host). Another management node is refused as a running site and copies from its backups. A container site's host must ship site_quiet_host and hold_container: it freezes the site from outside and holds the old container stopped once the copy has the site. A dead site's agent key must be on record, since the copy is installed as its copy. A site on a server of its own is refused while that machine serves anything else (machine_wide_refusals()): a site container its host report lists, a site its report names (sites), or another node on its address; its own test twin (<site>_test) does not count. A running one whose last host report does not list its sites (none yet, or older than 1.32) is asked for a status check first; a dead one's last report is all there is. The freeze, the power-off and an address swap act on the whole machine. The tab lists every reason that fails, under each of the two kinds.

From its backups (source_refusals(…, SiteCopy::FROM_BACKUPS)) asks nothing of the source's agent: not its words, not its reported recovery-key state, only its row, its layout, its release, its agent's key and this management node's backups of it, at any age (newest_chain($source, null)).

Every way the address could move is shown before the copy is made (SiteCopyMethods::for_target(), read-only): for each account a new server could be made in (at the region shown), for a brought server and for a host, the ways that can work and why the others cannot. The proxy where the domain's DNS is at a host that proxies (Cloudflare); an IP swap only between two servers made here at a provider that swaps addresses, in one account and one region, never a shared host's; the site's DNS records where the domain's host has a driver here (DnsDriverRegistry); and by hand always. An existing copy's ways are the same answer, with the swap judged from the two servers on record (SiteCopyRunner::switch_methods()).

The new server, one of two ways:

• Create it for me. The tab records an admin-origin CustomerCloudProvision in install mode copy (cvp_source_node_id, cvp_release = the source's release), in the cloud account chosen (any of ours, Linode or Hetzner, the default preselected, or a customer's connected one; HeldCloudAccounts::targets()), in the region and size chosen: each provider has its own pair of fields, shown for the account chosen, its sizes the plans its driver says the region shown sells (CloudInstanceTypes::instanceTypes($region)), the source's own plan chosen where it is at the same provider and the region still sells it, else the smallest with the source's memory. A plan the region chosen does not sell is refused before anything is made, naming the plans it does, as is a region that sells none. Its encryption is decided, never ticked (SiteCopyRunner::copy_encryption()), and said in one line on the form: in one of our accounts, encrypted by us, sealed to this management node's recovery key, with the provider's own disk encryption off; in its owner's account with the site's recovery key proven (its public half reported by the site and kept on its node, mgn_backup_recovery_public_key), encrypted by us, sealed to that key (cvp_recovery_public_key), the provider's off; in its owner's account without one, the provider's default, and the form says to prove a key first. One layer, never two: a region where the provider's own encryption cannot be turned off (a distributed region) is refused before anything is made. Nothing of ours while data_root_encryption_offered is off. The provisioning pipeline creates the instance and runs install_node in mode copy; the node row it makes is named <source> (copy), records mgn_copy_of_node_id, and a finished install leaves it in state copy (process_install_node), never working. The copy's agent then asks to join; the tab approves it after the provider confirms the instance runs at the join's address.
• I'll bring a server. The tab shows the commands to run as root on any fresh Ubuntu server (SiteCopyRunner::owner_command()): fetch the source's exact release from the release stream (utils/latest_release?version=, below), install the prerequisites, then install.sh site --bare-metal <site> … <domain> --enable-agent --management-node=… --dormant --copy-of=<id> --copy-of-key=<source agent key>. The owner reads the new server's fingerprint with joinery-agent status and approves the matching join request on the tab (it lists pending requests from an agent named after the source's site); approving makes the copy's row in state copy.

Root SSH keys. A copy is keyless with root login off unless the operator carries the source's keys. When the source's host report lists root SSH keys (root_ssh, from host_report.sh), the Create-it-for-me form shows each key's fingerprint and offers a tick to carry them. A key with options (command=, from=, restrict) is shown but never carried. The ticked keys are stored on the provision (cvp_root_ssh_keys) and the retire_install_password job installs them on the copy's root account; the start is refused if the source's keys changed since the form was drawn. The install form offers "Keep root SSH login" with a box for public keys on the same terms. With no keys, root login is off.

Either way the copy installs the source's own release, because vendor/ is never in a backup: its dependencies must be the ones the source's code was built for. A copy that has a newer release on its site is stale (SiteCopyRunner::stale_release()): a dormant copy's tab says so as soon as the site's release on record moves, Copy again and Switch over ask the site its release and refuse before anything moves, and the tab says to discard it and copy again. While a copy is being made, is switching over, or has switched and is not yet kept, the site's update waits in a staged rollout (below), so a copy never goes stale under its own run.

A copy run starts by itself once the copy's agent has reported its words, and again at each Copy again. One agent job at a time, each judged before the next is queued:

1. host_report on the copy, then a fit check from both machines' host reports: memory at least the source's, and free disk for the chain staged, the files unpacked and the database loaded twice over, plus 2 GiB (SiteCopyRunner::disk_needed(), an estimate from the chain's sizes);
2. backup_run on the source, at its own chain policy, so the copy is of the site as it is now, at the release the copy installed; the run copies the chain this backup is in (an increment of the newest chain, or a new one where the policy began one);
3. copy_export on the source, which waits for its owner's approval on the source's own Backups page (the tab links there and shows the copy's key fingerprint to compare);
4. copy_import, 5. copy_stage, 6. copy_restore on the copy;
5. site_census on the source, 8. on the copy, and SiteCensus::compare(…, false): informational, because the source is live and rows it wrote since its backup are expected to differ. A difference that cannot be explained that way (a sealed secret the copy cannot open, a canary that opens on one side only) stops the run.

The first step that does not do its part (a failed or refused job, an export declined or not approved in time, a step past its time budget) stops the run, names the step and why, and starts nothing after it. A run that passes leaves the copy dormant, current as of that chain's newest run, with the comparison and, from copy_import's result, the look link: one hosts-file line and the path that sets the copy's look cookie, so the owner can open the copy privately before anything switches.

Discard removes the copy's row from the dashboard, ends the record, and cancels the step it was on: a source waiting for its owner to approve the export sees the job withdrawn within half a minute and takes the request off its Backups page (agent 1.52.0+). The server itself is not deleted (the platform never deletes a cloud instance); the confirmation names it for the owner to delete at the provider, and any escrow key this management node holds for its encrypted data root is retired (DataRootEscrowSync::let_go()). A server being created or installed at that moment is left to finish first. A copy onto a host is removed from it by its host's agent (discard_copy, the job in scp_switch's discard_job_id), which refuses unless the host records the site as a dormant copy and the copy's own state file still says quiet copy; one whose install is still running is refused until it ends.

The copy is a tracked record, SiteCopy (table scp_site_copies), moved by SiteCopyRunner::advance() from the Advance Site Copies scheduled task every tick and when a node posts a copy step's result, under an advisory lock: a step that passes queues the next in the same call. The tab only reads, and reloads itself while a copy is moving.

Switching over makes the copy the site, from a dormant copy that is current, on the Copy tab (SiteCopyRunner, specs/site_copy.md steps 7–10). The tab offers each way the address can move (SiteCopyRunner::switch_methods()), with why one is not available:

• Through the proxy (proxied_origin): the proxied records in front of the site (Cloudflare's orange cloud) point at the copy, so visitors follow within seconds and no resolver cache is involved. Each press that touches DNS takes a DNS token for the site's zone in the form, builds the driver for that request (ProxiedOriginMove::driver_class_for(), from the domain's NS records) and keeps nothing.
• By IP swap (ip_swap, IpSwapMove): the two servers swap their public IPv4 addresses at their provider, so every record, allow-list and reverse DNS naming the site's address stays true. Both servers must have been created by this management node, at the same provider, one whose driver swaps addresses (CloudAddressSwap: Linode, and Hetzner's primary IPs), on one account and in one region; a provider that moves an address only between stopped servers (CloudSwapWhileStopped: Hetzner) has the copy powered off before the swap too, and on again after, each address move waited on until the provider says it succeeded; the provider credential is the one that created them (the account of ours they are in, or the connected account's grant, which needs ips:read_write and firewall:read_only). Two servers in different accounts of ours never swap. IPv6 addresses do not move.
• By the site's DNS records (dns, DnsRecordMove): every unproxied A and AAAA record in the zone naming one of the source's addresses (ProxiedOriginMove::machine_addresses(): for a site in a container, its Docker host's addresses too, which is where its IPv6 is known) changes to the copy's address of its family, with a DNS token for the zone typed at each press, at any DNS host this management node has a driver for.
• By the owner's own DNS change (manual), for every copy: the owner changes the records at their DNS host, and this management node watches them settle as it does its own change.

A copy onto a host is never swapped: its host's address is every one of its sites'.

A running site is frozen for its final copy, whatever the target (freeze_steps()): site_quiet on (its maintenance page up, its scheduled tasks held), a container source then held quiet from outside by its own host (site_quiet_host on, switchover), a final backup_run, the site's vouch, the copy's stage and restore, and both censuses, which must match exactly. The hosts are recorded in scp_switch's hosts, and a container source's old container in its hold, before the rows swap. The start of a copy onto a host is take_node_id, site_quiet off on the copy, site_quiet_host off on the copy's host, and provision_certificate filed on the host that issues for a container (certificate_issuer_for()). A copy whose certificate did not travel (a run from backups, or a container source, whose certificate is its host's) is issued one once its name points at it (copy_carries_certificate()). A container source's old container is held stopped last (hold_container stop), or, while a DNS move forwards visitors through it, once the move settles.

1. Switch over (begin_switch()). switch_refusals() first (read-only): the copy dormant and current, both agents reporting the switch-over's words (SiteCopyRunner::SWITCH_SOURCE_WORDS, SWITCH_COPY_WORDS; a copy from backups asks the source for none), the same release, the copy's look path known, and the source's newest backup still in the chain the copy holds the key for. Then the chosen way is checked, read-only, and anything that stops it is named with nothing frozen. By IP swap, IpSwapMove::plan() asks the provider: one public IPv4 on each server, Network Helper on for both (read from the configuration profiles, or the interface settings on the newer interface generation), every provider firewall of the source also on the copy, and no AAAA record of the site naming the source's IPv6; and, from this management node's own records (IpSwapMove::record_refusals()), a copy whose data root is encrypted must have an escrow holder, named in its last host report or this management node holding an enrolled key for the identity that report names (IpSwapMove::opens_by_itself()), since the swap restarts it. It records each machine's instance, address and reverse DNS, and the site's name. Through the proxy, with the token, ProxiedOriginMove::plan() reads the zone: every A and AAAA record naming one of the source's addresses (its host, its provision's addresses, and the addresses its agent reported at join, compared as addresses) must be proxied, the site's own name must point only at the source, and the copy must have an address of each record's family. Anything else stops here, naming the record, and nothing is frozen. The tab also says that a firewall letting only the proxy in is not visible from here.
2. The final copy (freezing). The source's row goes to switching, so no automation touches it, and one job at a time: site_quiet on on the source (the maintenance page, no scheduled tasks, no mail out), one more backup_run (a chain run on the longest interval, FINAL_RUN_FULL_INTERVAL_DAYS, so it extends the chain rather than starting one by age), copy_vouch on the source, copy_take_vouch, copy_stage and copy_restore on the copy, and both censuses, compared exactly: any difference stops the switch-over before anything moves. The final copy carries no secret and asks no approval: the copy already holds the chain key, certificate and DKIM keys from the last export its owner approved, and a frozen site's owner cannot reach its Backups page anyway. A final backup that started a new chain stops here: the copy holds no key for it, so the owner goes back and copies again. A dead source has nothing to freeze and no final copy: its server is powered off first when this management node created it, at any provider with a driver (a local power_off step), and the copy is ready. A container source shares its server, so the switch-over records its container and its host's node (SiteCopyRunner::old_container_hold(), the switch record's hold) and stops it later; switch_refusals() asks the host's agent for hold_container. Any other old server is the owner's to turn off, and the press says so. Every press that cannot be taken back asks in the system modal.

1. Move the address (ready, then move_address()). By IP swap, as local steps this management node does itself: the source's server off, the swap (POST networking/ips/assign, both addresses at once), each address's reverse DNS checked and put back if the provider dropped it, this management node's records following the machines (each provision's address, each row's host), the copy restarted so Network Helper writes its new address, and the copy's look path asked at the site's name (ProxiedOriginMove::prove()) for up to five minutes. A step that fails before the copy takes the node undoes the swap (undoing: the addresses swapped back, the copy restarted on its own, the source powered on, still frozen), and the copy is ready again with the reason on the tab. IPv6 addresses never move. Once the copy has started as the site, its IPv6 answers the site's name in reverse DNS when the site's AAAA gives it (IpSwapMove::name_copy_ipv6(); the owner points the AAAA there before switching), and the tab shows what happened (rdns_note), or, when no AAAA gives it, says to point the AAAA there and keep the switch-over; the moved IPv4 keeps the reverse DNS it brought. When the addresses swap back (an undo, or the way back), an AAAA record of the site still naming the copy's IPv6 is named on the tab, with the source's IPv6 to point it at. By the owner's own DNS change, the owner presses once the records point at the copy, and the look path asked at the site's name is the proof; unproven, the copy is ready again. A copy from backups has no certificate (none could travel from a dead server), so its proof asks without checking one (the look secret is the proof), and its start ends with provision_certificate on the node once the name points at it, then, for a container source, hold_container stop on the host: the old container stopped and kept stopped until it is removed. It is last, so the site, which already runs on the copy, never waits on it. A host that answers and does not stop it halts the start, naming it, and Try starting the copy again repeats the start. A host that does not answer (its agent not heard from within SiteCopyRunner::ALIVE_WITHIN_SECONDS, or the job not taken up within the step's wait) is not waited for: the step is owed, the switch-over ends and can be kept, and the stop stays queued on that host, which runs it if it answers again. The switch record's hold_owed says why, and the Copy tab says so, after the switch-over is kept too, until the stop has run (owed_hold_ran(), from the job's result or the task tick, which finds such copies by scp_switch's hold_owed key). Once the host has seen the old container gone (Permanently Delete Site, or its moved-site check reading absent), the task tick cancels a stop still queued, so a site installed there later under the same name is never the one stopped. Going back cancels it too, and starts the old container when its host had already taken the stop up (old_container_restarts()). Through the proxy (with the token again), the records are planned afresh and each is pointed at the copy (ProxiedOriginMove::move(), the address alone, so the proxy and the TTL stay; a failed write puts back the ones already written). Then ProxiedOriginMove::prove() asks https://<domain><look path> through the proxy for up to a minute, with a fresh query string each time: only the copy answers it, with a 303 that sets its look cookie. Unproven, the records move back and the site stays frozen, ready to try again or go back. By the site's DNS records, or by hand, the copy is first asked its look path at its own address under the site's name (ProxiedOriginMove::prove() with the name pinned to the copy's address), so nothing moves toward a copy that does not answer; then, with the token, the records change to the copy's addresses (DnsRecordMove::move(); a failed write puts back the ones already written), or, by hand, the owner has changed them. The start that follows (dns_start_steps()): the copy takes the node and starts; the machine answering the copy (its own, or its host for a container) believes the old server's forwarded addresses (trust_forwarder trust, for FORWARDER_TRUST_HOURS); the old server, now the retired row, forwards every visitor whose resolver still names it to the copy, over https, any other http request but the ACME challenge sent to https first (site_quiet forward, and for a container site_quiet_host forward on its host first); then a copy whose certificate did not travel is issued one, since the issuer's own resolver may be one that still names the old server.
2. The copy starts (starting). take_node_id on the copy, answered with the row swap; then site_quiet off on the node, which is now the copy's machine. The node is switched: it keeps its id, slug and history on the new machine, and the old machine's row is retired.

Local steps (SiteCopyRunner::LOCAL_STEPS: powering a server, the swap, the probe, the row swap back) take no agent job. A press that lays them out starts utils/advance_site_copy.php, a short worker that advances the copy every few seconds while its current step is a local one, so a machine's restart is followed in seconds rather than at the next task tick.

The way back (go_back()), offered from the freeze until the switch-over is kept, for every switch of a running site. When the address moved, it moves back first (through the proxy with the token, a record someone has since pointed elsewhere being left alone and named; by IP swap, the addresses swapped back, the copy restarted; by the site's DNS records with the token, the records back and their old TTLs with them, and the TTLs back even when the records never moved; by hand, the owner changes them back: the tab names each record with the copy's address and the site's own before the press (manual_records_back()), and afterwards, on the copy's ended banner, each one DNS still names the copy for). Before the swap, the site only starts again (site_quiet off, its host lifting a container's freeze first), and the copy stays a dormant copy, current if its final copy finished. After the swap, the copy's machine is quieted while it is still the node (and held off the network by its host, for a copy onto a host), the rows swap back, the copy's machine stops believing the old server where it did, the old server is powered on where a swap or a settled move powered it off (or asked to), an old container held stopped starts again, and the site starts again where it was. The copy is then discarded, with any escrow key held here for it retired: its agent took the site's node id, so it can no longer reach this management node. A copy onto a host is held stopped on its host instead (hold_container stop), with what was written on it. Its row stays on the dashboard, retired and named for it (<site> (copy's container)), and the tab says so: Permanently Delete Site on that row removes it from its host on the host's proof that the domain has left it, as it removes an old container. Anything written on the copy since it started does not come back, and the button says so in the system modal before it posts. A dead source's copy goes back only before it takes the node, and leaves the old server as it was: after the swap there is no running site to go back to.

Keep the switch-over (finish()) closes the way back, once a DNS move has settled (or its settle stopped): the retired row leaves the dashboard, its escrow key held here is retired, and the tab names the old server for its owner to delete at its provider. A switch by the site's DNS records asks the token once more, and puts each record's old TTL back (DnsRecordMove::restore_ttl()). A switch by IP swap whose copy's IPv6 is not yet named in reverse DNS asks again (name_copy_ipv6()), for an AAAA the owner pointed at the copy after the switch; still unnamed, the tab says to name that address alone with Set Reverse DNS (see Reverse DNS), so the moved IPv4 keeps its name. A container's old row is named instead, to be removed from its host with Permanently Delete Site, which proves the domain left rather than asking the unreachable site to approve (see Removing a container site). After a discard or a kept switch-over, the tab keeps naming the server to delete for a week, with what going back noted (an AAAA record still naming the copy's IPv6). A kept switch-over names the old server Keep named, which the copy records, so removing the old row afterwards changes nothing. A kept switch-over's old container is named on the site's Copy tab for as long as its cleanup is unfinished (SiteCopyRunner::cleanup_left()): until it is removed, and then while its host still reports a certificate for its domain, with Remove it from the host beside it.

The row swap (SiteCopySwap) moves only the machine columns (SiteCopySwap::MACHINE_COLUMNS: the agent key and what its agent reports, the host, the SSH fields) the provision links and the escrow rows (dre_mgn_managed_node_id: a machine's unlock identity is on its own disk), leaving the site's own columns on the node. The other row is named for what it holds: the site's name with (copy), then (old container) or (old server) once it is retired, and (copy) again on the way back. It is made only while answering the copy's take_node_id result (JobResultProcessor::complete_take_node_id(), from AgentChannelEndpoint::handle_result), which answers node_id_taken, and the copy changes its identity only on that answer, so a swap the copy does not hear about cannot happen. The copy's run moves on after the swap, in the same request. The old machine is not told: it goes on signing as the node, with the key that is now the retired row's (and after going back, the copy's machine signs as the node with the copy's row's key). While the switch-over is live, a request that fails against the node it names and verifies against the key of that node's copy's row is that row's (AgentChannelEndpoint::moved_machine(); the body's node_id is compared with the id signed as, and a job handed out is addressed to that id, the only one the machine accepts a job for), so the old machine still takes a DNS move's forward and, once it settles, its quiet, and the copy's machine its untrust on the way back. No other row or key is tried, and once the switch-over is kept or discarded nothing is.

Job Types

Note on bare-metal nodes: install.sh server disables root SSH during the bootstrap; the session that ran it is the last one the plane opens. Everything after it — backups, restores, certificates, upgrades — runs on the node's own agent, which is root.

One-Click Node Install

Dashboard → Install New Node opens a form that provisions a Joinery site in a single click. The Target Host dropdown offers three kinds of target:

• A known host (or Other server with manual SSH details): the form creates the ManagedNode and dispatches the install_node job immediately.
• Create a new cloud instance: no server exists yet. The form records an admin-origin CustomerCloudProvision (the cloud account — one of ours, the default preselected, or a customer's connected one — region, instance type, plus all install parameters) and the Provision Customer Cloud task births the instance, creates the node, and dispatches the install — see Customer-Cloud Fulfillment. The instance is created in, and billed to, the selected account; a Linode sign-in expires after two hours, so connect (or re-connect) shortly before submitting. In-flight provisions appear in a banner at the top of the dashboard.

Install types:

• Copy (install_mode = copy): a dormant copy of another node, made from that node's Copy tab rather than this form; see Site copy.
• Fresh: empty Joinery site with default schema. Admin picks the domain. The admin login is the username admin with a password generated for that site alone — there is no shared default. Like the generated Postgres password, it stays on the node rather than in the management node: read it at /var/www/html/{sitename}/config/admin_credentials.txt (root only), or set a new one with maintenance_scripts/sysadmin_tools/reset_admin_password.php. usr_force_password_change=true, so the first sign-in forces a new password.

The release stream (ReleaseStream::source_for()). A server takes its release, and its upgrades, themes and plugins after (--upgrade-server), from whoever serves this management node's releases: this node itself when it has published one whose archive is on disk, otherwise this node's own upgrade_source. A management node built from a release and never published from serves none (its ?serve-upgrade answers 404), so its servers go to the publisher it takes its own from, and run the same builds under the same key. A node that publishes and lacks a pinned version does not ask upstream for it: its numbers name its own builds. The join (--management-node) always names this node.

The mgn_install_state column tracks the lifecycle: installing → NULL (success) or install_failed (failure). A site copy (specs/site_copy.md) adds three more: copy (a dormant copy of another node's site, set when its join is approved onto a new row or, for a server this management node created, kept by a finished install of a row with mgn_copy_of_node_id), switching (the source of a switch-over in progress) and retired (the old machine after a switch-over, kept for the way back). ManagedNode::INSTALL_STATES lists all five with the words the fleet list badges them with, and ManagedNode::is_operational() is false for a node in any of them (or in any state it does not know): the fleet backups (FleetBackupPolicy::eligible_nodes, and with them the pruning that runs before each backup; FleetBackupRetention::prune also refuses one directly), staged rollouts (StagedRolloutRunner::node_refusal), the uptime task and NodeMonitorHealth::evaluate, the incident sources and the dashboard's status polling all skip it. The node stays in every list; its status dot is blue, a failed install red and a retired source grey. On failure, the node detail page surfaces a Retry Install button; the target must be cleaned manually (e.g. rm -rf /var/www/html/SITENAME) before retry because install.sh refuses to overwrite an existing site. Postgres passwords are auto-generated and stored in the target's Globalvars_site.php — Server Manager does not capture or display them.

Docker notes:

• The reverse proxy step is skipped when the domain is a bare IP address — a routable hostname is required for Apache ServerName-based virtual hosting. With an IP domain, the site is accessible directly on its mapped port, published on every interface. A site with a domain publishes its port on 127.0.0.1, reachable only through the host proxy.
• backup_project.sh requires rsync. The bare-metal and Docker install scripts install rsync as part of the essential packages (install.sh line ~948). Sites installed before this was added can install it manually with apt install rsync.
• After a Docker install, mgn_container_name is automatically recorded in the management node DB so future jobs correctly use docker exec to reach the site.

SSL Management

SSL State

Each node tracks its TLS certificate state in mgn_ssl_state:

Automatic Detection

A check_status on a node with an agent enumerates the certificate lineages the node holds, and JobResultProcessor matches them against the host this plane expects. Where no certificate is reported, an HTTPS probe from here catches a site whose TLS terminates at an edge. JobResultProcessor updates mgn_ssl_state and stores ssl_domain, ssl_expiry_raw, and ssl_expiry_ts in mgn_last_status_data. State transitions:

• CERT_FOUND → sets state to active (from any prior state)
• CERT_MISSING → clears state to null only if currently null or active; never overwrites pending or failed

Manual Provisioning

The Overview tab shows an SSL Setup card when mgn_ssl_state is not active, the node has a domain in its site URL, and mgn_cert_expiry_ts is empty. The last condition excludes nodes whose served certificate already covers their name (see Certificate monitoring) — their cert lifecycle is owned by an external renewer (e.g. Caddy), and the card's certbot-based provisioning does not apply to them. The card:

1. Resolves the domain via DNS and shows whether it points to the node's host IP
2. Enables the Provision SSL button when DNS is ready (or when the host IP is not configured)
3. On submit: starts the certificate chain (ProvisionPendingSsl::begin_chain), sets mgn_ssl_state = 'pending', redirects to the jobs tab

Cloudflare-proxied domains are gated on a routing probe first: the ssl_probe_place primitive writes a one-time token into the site node's webroot, the management node fetches /sm-ssl-probe.txt through the domain, and ssl_probe_clear removes it either way. The token is only fetchable because core serve.php routes that URL to views/sm_ssl_probe.php — a Joinery front controller never serves arbitrary webroot files, so a node whose code predates that route (PROBE_MIN_CORE_VERSION) is refused by name rather than blamed on Cloudflare. Only a match — proof that traffic for the domain actually lands on this node — dispatches the certificate; a miss leaves the domain pending until the customer's DNS actually routes here. The universal vhost already forwards X-Forwarded-Proto https, so nothing is patched.

Automated Provisioning (installs only)

For nodes installed via Install New Node, ProvisionPendingSsl (scheduled hourly) watches for nodes with mgn_ssl_state = 'pending'. It observes first: a machine this plane installs issues its own certificate during the install or on its host's retry timer, and the node's (or its host's) check_status enumerates every lineage under /etc/letsencrypt/live — a CA-issued, unexpired lineage covering the domain flips the node active with no job at all. Only then does it ask: it checks DNS and drives the certificate chain above on the issuer. After ~16 hours of failed certificate attempts it flips state to failed — except a Cloudflare domain still waiting on its DNS cutover (CF_ROUTING_UNVERIFIED), which never flips: a cutover the customer has not made is not a fault, and can legitimately take days. Instead the routing wait is paced — hourly for the first ROUTING_FAST_ATTEMPTS tries, then one try every ROUTING_SLOW_GAP (six hours) — and entering the slow lane emails the operator once (recipient chain: server_manager_provisioning_admin_alert_email → webmaster_email → first superadmin; the sent marker is stamped into a job row's parameters as routing_alert_sent). The slow lane and the alert apply only while the domain still resolves to Cloudflare: once it repoints, the next attempt is due within the hour, and the 16-hour give-up window opens fresh at the first non-routing failure — time spent parked at Cloudflare never burns it. Manual provisioning via the Setup card is the fallback.

Hosting Provisioning

Paid hosting orders on getjoinery become installed, SSL'd Joinery sites with no human touch. The Poll Hosting Orders scheduled task polls the getjoinery API each cron tick for paid orders carrying an answer to the configured domain Question, and fulfills each one in the mode the product declares in pro_fulfillment_provider:

• Shared host (the default, any other value): the pipeline picks the least-loaded provisioning-enabled ManagedHost, assigns the next Docker port, and dispatches an install_node job. The buyer's site is a container on infrastructure the operator owns.
• customer_cloud: the buyer's site runs on its own cloud instance. A product opts in by picking Customer cloud server in the product-edit Purchase grants picker (CustomerCloudFulfillment, registered with the store's FulfillmentRegistry from serve.php); that stamps the provider value and contributes the one checkout requirement the line carries — the id of the site the buyer configured beforehand (see The buyer configures first below). The reference picked beside it decides *whose cloud account the server is born on, and it is the product's decision rather than the buyer's: - the buyer's own account — they connect it, the provider bills them, and none of the operator's keys are involved. See below. - the operator's account (hosted) — the operator's own token creates it, there is no Connect page, and the mail, billing and banner legs apply. See Hosted tier.

Nothing about a customer-cloud site is asked in the cart. The buyer describes it on one page, /profile/server_manager/configure (signed-in only; the Managed product page sends a buyer with no configured site there). A visitor with no session is sent first to /server_manager/start, step 1 of three: the account, offered as sign-in and sign-up side by side with no preference. Both forms post to the start page, which hands them to login_logic and register_logic — so every gate those hold (throttles, activation, the second-factor divert, the bot defences) holds there — shows a refusal in place, and lets every success land on the configure page through the session's return slot. Every page on the path carries the step strip (ManagedSiteDraft::steps_html(): 1 Your account, 2 Your site, 3 Payment; the summary, whose one button is Continue to payment, is step 3). The configure page asks for: the domain — their own, or one this plane registers for them, with the live availability check and price and the registrant contact block — the site's internal name, the region (server_manager_hosted_regions; one region in the list means the field does not render), and the site admin email. Save and continue validates everything (ManagedSiteDraft::save_and_freeze, with the domain gates in ManagedDomainIntake) and writes a draft provision: a cvp_customer_cloud_provisions row with cvp_origin = buyer at status pending_payment, carrying the domain source (cvp_domain_source: own | register), the frozen domain-year quote (cvp_domain_quote, _time) and the sealed registrant block (cvp_registrant_sealed). A draft holds nothing — no instance, no domain, no slug — and the pipeline cannot see it (draft and pending_payment are outside every phase's query).

Continue to payment is a single-button form posting product_id, product_version and managed_site=<draft id> at the product's own URL, so the store's product_logic adds the line exactly as a product-page submit would. ManagedSiteRequirement — contributed by `CustomerCloudFulfillment:: extraRequirements()`, never attached by hand — is the whole coupling: it renders no fields, validate() refuses anything but the signed-in buyer's own pending_payment draft, process() stores the draft id (managed_site) and the freeze time (managed_site_frozen) in the line's form data plus, when registering, the frozen quote under the keys the registration guard reads (managed_domain_price_line, managed_domain), and extra_cart_lines() adds the domain-year line from that frozen quote without ever re-quoting.

The draft's states:

A management node that holds a cloud account flagged test (see Cloud accounts) cleans each such account up on its own, one at a time. The Delete Old Test Servers task (TestCloudCleanup) runs hourly and is off until turned on under Scheduled Tasks. It deletes servers, and storage volumes attached to nothing, that are older than server_manager_test_cloud_max_age_hours (default 24).

The safety catch is the account's own name. Every run reads the account's company name from the provider (CloudAccountCleanup::accountCompany(); on Linode, Account Settings, which needs account:read_only). It deletes only when the name is exactly Joinery Test disposable. On any other account, or when the name cannot be read, it deletes nothing on that account. It then records the refusal, and plane:test_cloud_cleanup opens on this management node's own node, naming the account and the company it found. The incident clears when a run passes or the task is turned off. A live account flagged test by mistake, or the task turned on at a production management node, therefore deletes nothing. No site setting can make an account count as disposable.

Never deleted:

• anything tagged keep at the provider (any case);
• a server at the address of a live managed node here, IPv4 or IPv6, by the transfer watch's address rule. It is listed as held, so the operator tags it keep or removes the node first.

1. Job agent. Every job the pipeline creates sits pending until a joinery-agent polling this site's queue claims it. Install the agent on the management node's host (`sudo bash joinery-agent-installer.sh --config <Globalvars_site.php path>`, built from the agent repo's build_installer.sh); hosts without systemd are detected and supervised via cron. The page's agent badge must show Online before anything below can execute.
2. Per hosting product. For customer-cloud fulfilment, pick Customer cloud server under Purchase grants; that stamps the provider, and the buyer configures the site on /profile/server_manager/configure before paying — nothing is attached to the product by hand. For a buyer-account product, put the Connect link (https://<management-node-host>/profile/server_manager/connect_cloud) in the after-purchase message; the Connect page is deliberately in no member menu. For shared-host products, attach the domain question as a requirement instead — the attachment is what makes an order a hosting order. To sell domains on the configure page, see Managed domain registration below.
3. Shared-host fulfilment only. Opt at least one managed host in from the dashboard (Edit → Max Sites + Provisioning Enabled). Its IP is sent to customers as the DNS A-record target, so it must be routable.
4. Buyer-account fulfilment only. Register the OAuth client in Linode Cloud Manager (Profile → OAuth Apps → Create OAuth App, not public, callback https://<management-node-host>/oauth_callback) and enter the client id and secret at Admin → System → OAuth Providers. The referral URL from Cloud Manager → Profile → Referrals goes in the page's field.

On the dashboard, a kept box's row shows its holder Reveal root password (shown once, on a page of its own) and Retire; above the list, Retire all requested (boxes whose owner turned it off) and Retire all kept; the box's node page (its Emergency root access box, on the site's node and on a Docker host's own node) offers its holder Reveal and Retire too, answering there. Each needs the viewer to be the row's holder, their vault open and their second factor used recently; a holder who has not confirmed it in the last five minutes is offered Confirm your passkey to reveal or retire in their place, and comes back to the buttons. Retire seals the password back under SecretBox and returns the row to held with cvp_root_access_offered cleared, so the next pass retires it and the box is never kept again. Every Reveal and Retire is a row in the general event log (break_glass_reveal, break_glass_retire, naming the provision and node), listed on the box's dashboard row and on its node page's Emergency root access box. Nothing removes a kept password unattended. plane:break_glass_attention (warning) lists kept boxes whose owner asked, whose node is gone, or that are sealed to someone other than the current holder; plane:break_glass_no_holder (critical) is raised while the setting is on and the holder is unset or unfit. Both are raised on this management node's own node.

The plugin declares vaultConsumer.reseals, and its bootstrap registers the re-seal of CustomerCloudProvision rows, so a rotation of the holder's key re-wraps kept passwords. A key rotation refuses on a management node where that callback does not load.

A join from a provisioned machine is checked with the provider. The join records the client address this plane can vouch for (behind Cloudflare, the client header only from a verified edge). When that address is a provision's instance, the join card names the provision, instance and age; approval asks the provider for the instance and refuses unless it is running at that address and the node being approved is the provision's site or a host record at its address. A second join from that address after the node's agent is admitted is refused and logged as an alarm. A docker provision's container gets mgn_mgh_managed_host_id (its placement record, minted at booting), and the host's own agent, once approved, is named in mgh_mgn_managed_node_id. The server is the customer's property: cancelling their subscription stops management, never touches the instance.

The install itself runs from this management node, not from a node agent: a machine we just created has no agent yet. ManagementJob::createJob creates every bootstrap job (ManagementJob::BOOTSTRAP_JOB_TYPES: install_node and retire_install_password) in status queued, which the agent channel's claim (pending only) never matches; the Run Install Jobs scheduled task spawns plugins/server_manager/utils/run_install_executor.php (detached, single instance, log at logs/install_executor.log), and InstallJobExecutor claims each queued job, runs its local steps here and its ssh steps on the target over the sealed root password (sshpass, password in the environment, never on a command line), and writes the same output and status contract the agent runner writes. Every shape travels it — fresh, copy (a dormant copy of another node's site) and bare (a Docker host with no site) — as one session (specs/ssh_single_bootstrap.md). Fleet enrollment seeding (the mailbox plugin's FleetProvisionSeeding) is the fleet_enroll primitive on the new site's own agent, so it waits for that agent to pair: cvp_fleet_seed_state goes pending at completion, dispatched once the node reports the primitive, then done or failed by the job's answer.

One layer of encryption, never two. A server whose data root is encrypted by us has the provider's own disk encryption off; every other server keeps the provider's default. cvp_data_root_encrypted decides it, set when the provision is created and only where the core setting data_root_encryption_offered is on (off by default): a hosted order or an activated Managed draft on our account, Remote Install into one of our accounts, and a site copy onto a new server, whose layer the Copy tab decides from where it is born (SiteCopyRunner::copy_encryption()): in one of our accounts, ours; in its owner's account, ours sealed to the site's own proven recovery key (cvp_recovery_public_key), or the provider's default where none is proven. Any other server in a customer's own account is born unencrypted. For such a provision the task refuses, before anything is created, when the key it is sealed to is not proven (this management node's, unless the provision names the site's own) or the region is a distributed one (where Linode's layer cannot be turned off); creates the instance with disk_encryption: disabled (sent only where the region lists Linode's "Disk Encryption" capability) and swap_size: 0; saves cvp_instance_id, then records the provider's answer in cvp_provider_disk_encryption (enabled, disabled, or unknown when it gave none) at create and again at booting. A layer reported on fails the provision before anything is installed, leaving the instance for its owner to delete; unknown proceeds. The install then runs install.sh docker or install.sh server with `--encrypted-data-root --recovery-public-key=<that key>`. An adopted server records what the provider's listing said and is never marked encrypted by us; relays are born with the provider's defaults. The node page says it in one line under the address: "Data root: encrypted by us · provider's disk encryption: off" (or on, or unknown).

Encrypting in place keeps to one layer too. The node page's Encrypt the data root refuses a machine whose provider's layer is on, except a management node, which never copies itself and may keep both. What it says to do instead is DataRootEncryptionOffer::provider_layer_fix(): copy the site onto a new server, into its owner's account once the site's recovery key is proven where it is not. A site's own Backups page asks root to encrypt in place and cannot read the provider, so this management node records the layer on each machine (the data_root_provider_layer word, sent by DataRootLayers::push_provider_layer whenever a host report's data_root_provider_layer says otherwise than it knows); joinery_data_root.sh encrypt then refuses there as well, unless --keep-provider-layer, and the page says to copy the site instead.

The escrow holder. This management node holds the escrow secret of each encrypted node enrolled with it (dre_data_root_escrows, EscrowEndpoint at /api/v1/agent/escrow/*) and answers its boots; the node page's encryption card shows what it holds, the last answer and refusal, the addresses a node may ask from, and Unlock it from here with the recovery key in the browser. A machine that is not one of its nodes asks to enrol on the dashboard's Machines asking to have their key held here, where a person compares the fingerprint. For a node in one of our accounts whose key holds events:read_only the holder is Tier 2 (EscrowTier2): it reads the provider's activity log before it answers, refuses a boot after a rescue boot, a root-password reset and the rest, and offers I did this myself or a copy from backups. The full picture: Encrypted data root.

Settings: server_manager_customer_cloud_region / _type / _image (instance defaults), server_manager_linode_referral_url. Provider credentials are the core oauth_linode_* settings (Admin → System → OAuth Providers).

The compute API surface is CloudComputeProvider (includes/cloud_compute/) with LinodeComputeDriver implementing it; a new provider is a new driver plus its OAuth provider class.

Managed Domain Registration

A hosting buyer who does not already own a domain can buy one in the same click as the server. At checkout they type the name they want, see live availability and the one-year price, and fill a contact block prefilled from their account. One payment covers both. Behind it the pipeline registers the name, points it at their box, publishes their mail records and sets reverse DNS — so their website answers and [email protected] works by the time the welcome email lands. No registrar dashboard, no DNS panel, no waiting on the buyer to paste a record somewhere.

The buyer legally owns the domain from the moment it is registered. They are the WHOIS registrant on day one; the operator holds only management and billing, so that buying it could be one click. Those move to the buyer later (see Graduation). Ownership is never in question and never waits on a step.

Selling it

Two things have to be set before a domain can be sold, and until both are the checkout field refuses the order rather than taking money for a name it cannot register:

1. Registrar credentials — the Domain registration card on Server Manager → Provisioning. Namecheap needs an API username, an API key (sealed at rest), and the management node's public IPv4 address allowlisted in its API panel. Namecheap grants API access only to accounts with 20+ domains, $50 in the balance, or $50 spent in the last two years. A sandbox switch points every registrar call at Namecheap's sandbox for a full rehearsal.
2. A domain-year product — an ordinary store product, not publicly listed, with one version whose price type is user. Select it in the store's store_domain_registration_product_id setting. Its price comes from the live registrar quote at checkout, so the buyer pays one year at cost.

The answer therefore lands on a later tick than the question, and the mail step is a four-state check rather than a call: a completed and unread job is read, published and marked consumed; a job in flight is waited for; a finished one is re-asked after PREPARE_RETRY_GAP_MINUTES. Reading it exactly once is what makes the incomplete answers work. A record set returned without DKIM is published anyway (MX and SPF are what make mail arrive) but the step stays open until the signing key is included — and without the consumed mark, that same answer would be re-read and re-published every tick and the signing key never asked for again. The lookup is scoped to the domain as well as the node, because a shared host carries many managed domains. A node whose agent does not offer the primitive writes the reason onto rdm_error, where the Domains page shows it, and is retried rather than failed.

Before anything is bought, the order is checked for the money. The checkout answers and the payment are two separate objects, and the cart lets a buyer separate them: every cart line carries its own Edit and Remove, so the domain-year line can be deleted — or repriced through its own product page — while the hosting line carrying the answers is submitted unchanged. The intake reads the hosting line, so without a check the domain would be registered on the operator's card for free. The rule: the order must hold a paid domain-registration line worth at least the quote, and each such line backs at most one registration. Anything else parks the row with an alert. That turns every one of those doors into something an operator sees rather than a silent loss.

Publishing always goes through DnsReconciler in additive mode, never a driver's raw call: Namecheap's setHosts replaces a zone's entire host list, and additive means the pipeline can create records a zone lacks but never overwrites something a person put there. A shared-host row stamps the PTR step immediately — one address serving many domains has no per-domain PTR to set.

The mirror case has its own sweep, because there is nothing to run it from. A buyer who removes the hosting line from the cart and keeps the domain line pays for a domain year whose intake never fires — no row is written, so no queue could ever show it. ManagedDomainWatch therefore looks for the arithmetic signature directly: an order with more paid domain-year lines than registration rows. It reports each such order once (a high-water mark over order-item ids), and gives a fresh charge fifteen minutes to file its row before judging it.

A terminal failure parks the row at failed and emails the provisioning alert address. It is never auto-retried: a name someone else took needs a conversation with the buyer, not another attempt.

Ownership and graduation

Legal ownership is immediate and never moves. What moves is custody — rdm_graduation_state, running operator_managed → push_requested → push_sent → self_custody.

While the domain sits in the operator's registrar account, its renewal bills the operator, and the platform never renews a buyer's domain and never fronts the cost. So the domain has to reach the buyer's own account before its first expiry. ManagedDomainWatch is what makes sure it does:

• It sweeps for domain years that were paid for but never registered (above).
• It refreshes the expiry from the registrar at most weekly.
• At expiry minus six months it pushes a custody state to the buyer's box, and ManagedDomainNotice starts rendering a take-ownership notice there — calm at first, sharper at 30, 14, 7 and 1 days. That notice is the buyer's first mention of graduation anywhere; nothing in the setup wizard or the welcome email raises it. It is shown to permission-5+ users only, never to the site's visitors.
• Once a push is in flight it asks inAccount(). False is the success signal — the domain has left the operator's account — and flips the row to self_custody, updates the box, and emails the buyer a confirmation with the auto-renew reminder.

They are written by a managed_domain_notice job, and the setting names are not on the wire. The job carries four VALUES; which settings they land in is decided by utils/managed_domain_notice.php on the node, which writes them through Setting::put — so an undeclared name is refused by the declared- settings gate, and a generic write-a-setting job that could reach the rest of stg_settings does not exist.

ManagedDomainWatch converges on that state rather than pushing at it: each tick computes what the box should be holding, compares it against the last notice job that completed for that node and domain, and files one only when they differ. A push that failed therefore self-heals on the next tick, and rdm_prompt_pushed_time — the record that the buyer has seen the take-ownership notice at all — is stamped from a job that COMPLETED carrying a state that renders one, never from a dispatch.

Adding a registrar

DomainRegistrarProvider (includes/domain_registrar/) is the seam; DomainRegistrarRegistry discovers implementations by interface, so a second registrar is a class in that directory and nothing else. It covers availability and price, registration with a registrant contact, WHOIS privacy, expiry, a custody probe, and which DNS driver serves its zones. It has no renewal call and no DNS methods — the platform never renews, and records are published through the shared DNS stack by driver key.

Joinery-run services for self-hosted sites

A self-hosted site can send its email through this plane's SMTP2GO account and keep its offsite backups in this plane's backup storage, on credentials cut to its own slice, for as long as a paid-through date on this plane says so. The customer's own Managed site uses the same legs because that is what Managed is; a self-hosted site rents them, and the difference is only how the credential reaches the box. The master keys never leave this machine.

The account link. Nobody types a key. The site's setup wizard (Email or Backups step) shows a Connect your getjoinery account button; it mints a single-use state bound to the owner's browser session and sends them to /services/authorize?site=&return=&state= here. Signed out, they go through the platform's sign-in (sign-up beside it) and come back; signed in, the page names the site and the account and takes one click. Approval is a POST that mints an ApiKey for the site (ServicesConnect::mintKey: machine, read + write, no delete, named Joinery services — apk_name is varchar(32), so the host is recorded on the tenant rows, not the key) and redirects to the site's own return address with the pair and the state, once. The return must be on the site's host over https. One active key per site per account: a re-connect mints a new key, moves the site's rows to it and deactivates the earlier one. Connecting is the site's first contact: one ServiceTenant row per service is created at unpaid.

The tenant row (svt_service_tenants, ServiceTenant) is one site's standing with one service (mail | shelf), keyed by the connected key (svt_apk_api_key_id); the host is a label refreshed on every status call. What decides is svt_paid_until: null means never entitled. States: unpaid (no date yet), provisioning, active, suspended (the date passed and the grace ran out), released (the customer left, or the account holder disconnected). The four shared states are the core ServiceTenantLadder's vocabulary. The figure (svt_figure) is sends this month for mail — the provider's month-to-date count, read hourly by the reconcile, nudged between reads by the SMTP2GO webhook — and the ledger's completed bytes for backup storage.

The actions the site calls over its key, all under /api/v1/action/server_manager/: services_enroll (service, host), services_status (host), services_release (service), and the five shelf broker actions below. JoineryServices is the operator side of all of them.

• Mail enrol builds everything at once (Smtp2GoLeg, shared with ProvisionHostedMail): the subaccount capped at server_manager_hosted_send_allowance, the sender domain (the host itself) (zero DNS records from the provider fails loudly), and one SMTP user. The answer carries the send values (host, port, username, password, sender, HELO, hostname) and the DNS records; the site writes them and publishes the records. The password is never kept on this plane: a second enrol removes the tenant's SMTP user and mints a fresh one in the same subaccount, so the answer is always a working credential. A host change adds the new sender domain and releases the old. The status answer reports domain_added until the provider verifies the domain, probed once per status call.
• Shelf enrol mints nothing: the answer is the tenant's slug, backup storage path prefix, {prefix}/{slug}/, the 90-day retention promise and the bucket's coordinates — no credential anywhere in it.
• Status is umbrella contract C2 per service: figure, allowance, paid_until, state, notice, the first door (action_label, action_url — the referral settings server_manager_smtp2go_referral_url and server_manager_storage_referral_url, '' when unset), manage_url (the account's Connected sites page), plus the ready-made banner row (label, used_label, allowance_label, percent).
• Release removes mail's SMTP user and sender domain from its subaccount and closes it now (a removal that fails keeps its name on the row, and ServiceTenantWatch reopens the subaccount, removes what is left and closes it again until both are gone), or marks backup storage tenant released so the broker refuses it; backup storage is kept 90 days from that day (svt_prune_after_time) and then pruned. Idempotent. A new paid-through date on released mail waits for the site's next enrol, which reopens the subaccount, adds the domain and mints a user.

The reconcile is the Services phase of ServerManagerAdvanceProvisioning (ServiceTenantWatch), last, every tick. Per tenant row: the allowance is re-read and a changed one re-sets the subaccount limit; the date is compared every pass and the row walked down the ladder — server_manager_services_grace_days (14) after the date passes the row is suspended (mail's subaccount closed; the backup storage broker refusing) and the retention clock starts, and a new date at any point before the prune reactivates in place; an act the provider refused (unreachable) is retried next pass. Mail's figure is read from the provider hourly. The backup storage ledger is reconciled daily, space by space, against a listing of that space's own target (an object it lacks is marked pruned, a multipart still open at the provider aborted first; one the ledger lacks is adopted at its listed size; a space that cannot be listed is reported and left alone); a run open longer than 36 hours is aborted on the plane's side, its open multipart cancelled with the plane's credential; every active tenant's backup storage is pruned to the newest server_manager_services_shelf_keep_chains (4) chains per profile across its spaces, chains whole, each from its own target, a chain with an open run never touched, and only as BackupSafety allows (the newest verified chain and everything newer stay, the newest stays, and a chain goes a day after a pass first found it surplus — see What retention will not delete in docs/backups.md); a draining space is kept whole until the active space holds a run the site reported verified, then retired once empty; a suspended or released tenant whose prune-after day has come loses everything in its spaces once, and the row says so.

Node-linked rows: a site moved off Managed. A hosted site handed to its customer's own Linode account (Hosted tier § Moving a site) keeps its mail and backups through two rows linked to its node (svt_mgn_managed_node_id) instead of a key — it never ran the Connect flow, so the site itself cannot call enrol, status or release. Only the reconcile and the operator reach them. The mail row is the Managed leg's subaccount, SMTP user, sender domain and records, carried over; from there the meter, suspend and close work as for any tenant. Backup storage is the node's own fleet backups in its storage spaces, not the broker: the figure is what they occupy (mgn_backup_shelf_bytes), the allowance pauses them and lifts its own pause, suspend and release switch them off with a marker and reactivate switches them back on (NodeBackupShelf), and the prune empties what this management node took in each of the node's spaces ({space}manager/), never the site's own backups beside them; the ledger, stale-run and chain-retention passes do not apply. With no status poll on the box, the reconcile composes the site's services banner from the node's rows and pushes it over the agent when it changes, and the customer gets one email when grace starts and one at suspension (services_moved_grace, services_moved_suspended). Connected sites shows such a site as moved from Managed with no Disconnect — one click would close its mail; the customer leaves our mail through their own wizard's Email step, then the operator releases the row.

Operator pages. The Service Tenants page (/admin/server_manager/service_tenants) lists every row with state, figure, date and the ladder's timestamps, and carries the two acts: Grant (write the paid-through date — what entitles a tenant in this phase; a stopped row with a date ahead comes back in place now, an unpaid row waits for the site's next enrol) and Release. Managed sites have no row and never appear; a moved site's node-linked rows do, naming their node. The account holder's Connected sites page (/profile/server_manager/services) lists their linked sites with each service's state, date and figure, and carries Disconnect, which deactivates the site's key and then releases every service it holds; a provider refusal on one release is surfaced after the rest have run, with the key already off.

Removing an account waits for its services. An account is removed for good only once none of its tenant rows holds anything at a provider (ServiceTenant::provider_holding()): a mail row whose subaccount is still open, or a backup storage row whose copies are not yet pruned (copies in its storage spaces, or a moved site's fleet backups while its node is still there), refuses the removal whole, and the dry run names the site and what it holds. Release the row first (the account's Disconnect, or Release on Service Tenants); a backup storage row then waits out its 90 days, the prune empties its spaces, and the removal goes through, taking the rows and their spaces with it. The removal itself calls no provider.

The webhook (ajax/smtp2go_webhook) maps a delivery to a Managed site by its SMTP username first; one it cannot map is tried against the tenant rows' usernames and moves that tenant's figure.

Backup Targets

Backup targets define where the backups this management node takes, and the backups it keeps for customers, are stored. Nothing is inferred from "the one enabled target".

Storage spaces

Every stored backup on this management node is in a storage space (sps_storage_spaces, StorageSpace): one owner's folder on one target — a node's ({target folder}/{node slug}/) or a backup storage customer's ({target folder}/t{id}/). Every reader reaches a backup through its space, never through "the target in use now".

Moving. Move on a node's Backups tab, or on a customer's row of the Service Tenants page, sends the owner's new backups to another switched-on target: that target opens (or gives back) the owner's space there and the old one starts draining (StorageSpace::move()). Move everyone on a target's page does the same for every owner whose new backups go there (move_everyone_off()). Nothing is copied: the next backup starts a full one on the new target — the broker takes the run in the owner's new space, the node starts a new chain when the space changes, and the broker refuses to extend any chain whose objects are in a draining space — and the old backups stay readable and restorable where they are until they age out.

JobCommandBuilder::node_space() returns a node's space by id, or its active one; get_target() is the active space's target switched on or off; write_space() / write_target() return it only while its target is switched on, for new backups and uploads. Downloads, Prepare, Verify and Bring them back name the space a backup is in (space_id, from the listing), and a chain named without one is found in whichever space holds it.

Supported Providers

Every provider in StorageProvider (Backblaze B2, Amazon S3, Linode, Cloudflare R2, Wasabi, DigitalOcean Spaces, Hetzner, generic S3-compatible). The form is BackupTargetForm, the same one the core Backups page draws; each provider shows the region and endpoint fields it asks for, and Backblaze's are read from the key at save time. A target's location, how it is switched off and when it may be deleted follow Backups § Saving a target.

All providers authenticate against their S3-compatible endpoint via AWS SigV4 signing performed by S3Signer.php. There is no per-provider CLI dependency — uploads, downloads, deletes, and listings all run as direct HTTPS calls, from the management node (web tier) or from the node's own agent.

Configuration

1. Go to /admin/server_manager/targets and click Add Target
2. Select a provider, enter bucket name, path prefix, and credentials
3. Choose it under Where new backups go, so new nodes are given it
4. For a node that already exists, choose the target under Move new backups to on its Backups tab

{"access_key": "...", "secret_key": "...", "region": "...", "endpoint": "..."}

• Scan for Backups — creates a list_backups job to scan local /backups/ on the node
• Unified file table — shows filename, size, date, and location (Local / Cloud / Both)
• Upload to cloud — offered on rows that exist only on the node, when the node has an enabled cloud target. Creates an upload_backup job that pushes that one file from the node to the target. The transfer runs on the node, where the file already is; routing it through the management node would drag the archive down and push it straight back up. The local copy is kept regardless of the node's delete-after-upload setting — an operator asking for an offsite copy of a file they are looking at did not ask for that file to disappear, and deleting stays an explicit action. The button waits for the job's real verdict, so a failed transfer reports as failed with a link to the job output rather than reading as done
• Delete — single Delete button per row that removes the file from every location it exists in (local, cloud, or both); the confirmation dialog names the file and locations explicitly
• Restore Full Project — for .tar.gz archives, see the restore_project row in the Job Types table
• Backups — one row per backup run in the node's backup storage, newest first (when, full or incremental, who took it, size), with the last backup, the last full backup and the oldest backup held stated above the list; read from each chain's manifest.json by BackupChainListHelper, which lists the node's own folder in every live storage space, every page of it (a run kept on a target the node was moved away from says so), so ten thousand offloaded-file objects under one node never push a manifest off the end of the list. Restoring a run replays the last full before it and every incremental up to it, in order. Chain artifacts are deliberately absent from the flat file table above — listed there, files-0003.tar.gz.enc invites a restore of one incremental with no full under it, which restores nothing at all. Last verified restorable is stated with the three facts (level name, date, the node's own counts, or the reason it failed), and each run's Verify button opens a dialog with two choices, Open and read and Rehearse a restore (needs about N free on the node), the room worked out from the runs' recorded sizes the way the node works it out before downloading. Either creates a verify_backup job; the poller reports what the node said, and a reload shows it above and on the node's card. An Offloaded files in backup storage row totals the object store from the same listing — "N objects, X GB by this management node; last indexed at the run of …" — and, when a run in backup storage carries an offloaded-files index, offers Bring them back: the restore_objects survey job in missing mode, whose pages follow as their own jobs; the node's own Backups page says what its file store is missing, the plane only asks

• recovery — the recovery public key the node itself holds and has verified, read on the node. The private half lives in a password manager, held by whoever administers that node, and never touches a server. An operator who administers several sites may configure the same public key on all of them, and then one private key opens all of theirs; that is their arrangement to make, on each site, and not something this management node can impose from here.
• site — a keypair the node itself holds at config/backup_site_key. This is what lets a site restore itself with nobody present: pre-restore rollback snapshots and routine restores need no operator. It is disposable — lose it and the recovery key still opens everything, and the next run mints a new one.

• The recovery keypair is generated with maintenance_scripts/sysadmin_tools/escrow_keypair.php (standalone PHP + sodium, no platform bootstrap, so it runs during disaster recovery when the platform is gone), or in the browser from the setup panel.
• The public key is stored in the core backup_recovery_public_key setting.
• Minting and sealing happen on the node, in maintenance_scripts/sysadmin_tools/backup_envelope.php. Only the recovery public key travels in the job step, so a ManagementJob row — which persists forever — carries nothing that can open anything.
• The plaintext key exists only as a 0600 file for the length of the run and is shredded by the step that seals the envelope to the finished archive.

Pruning is driven by a bucket listing, which is the opposite of what a site does for its own backups, and correct only here: this management node defined the whole {prefix}/{slug}/manager/ path, knows every slug under it, and is the only party that can delete from it. It is also stricter — it keeps only sets of objects that actually exist, so a run that failed part-way can never be counted as a restore point. Chains are grouped by their directory, so they are kept or deleted whole by construction. How long is the site's choice: each backup run reports the site's backup_retention_days as BACKUP_KEEP_DAYS, the result processor stores it in mgn_backup_keep_days, and the pass keeps that many days — never fewer than the policy's keep_days (server_manager_fleet_backup_keep_days, 7 by default), the floor that stops an intruder on a site from shortening its window to erase its history. A node that has not reported a window is read at the site default of 28 days (FleetBackupPolicy::retention_days()). Within the window the rule is the one a site's own retention uses (BackupRunner::surplus()): every group started inside it, dated by the stamp in its name, and the newest one started before it. What that finds surplus goes only as BackupSafety allows: the newest chain a verify_backup job passed (FleetBackupRetention::verified_chains()) and everything newer stay, the newest stays, and a point goes a day after a pass first found it surplus, recorded on its storage space (sps_surplus; see What retention will not delete in docs/backups.md). The site removes its own records of manager runs outside its window by the window alone, so its Backups page lists what is kept.

Every run of a chain writes its own manifest (manifest-{seq}.json), so no key in backup storage is written twice and a versioned bucket keeps no superseded versions of one.

Scheduling

The Fleet Backups task (plugins/server_manager/tasks/FleetBackupRun.php) runs every cron tick, finds due nodes, prunes each one's backup storage across its storage spaces (FleetBackupRetention::prune(), each point deleted from its own space's target; a draining space kept whole until a verify of a chain in the active space has passed), and dispatches one backup_run per node.

FleetBackupPolicy resolves each node's schedule: the declared fleet settings, then that node's own mgn_backup_policy overrides. The fleet default is enabled. That default is what stops a newly managed node being forgotten — there is deliberately no detector for "nobody has decided about this node", because a node nobody decided about is backed up anyway.

The node detail Backups tab edits the policy, as one of three positions:

• Fleet default stores nothing, so the node follows the fleet settings — including future changes to them.
• A schedule of its own stores the full field set (frequency, window, mode, retention, full interval, days between verifications), frozen against the fleet default: a value the operator saw and saved is a value they chose.
• Off stores exactly that decision, which is what lets the dashboard treat a node without fleet backups as somebody's choice rather than a gap.

Due is keyed on when the last run was started, not on whether it succeeded. Retrying a failing node every fifteen minutes until its next slot would hammer a machine that is already unwell.

Verifying what was backed up

The same pass proves the backups it takes, at two of the three levels described in Verifying backups:

• The backup storage check runs on every retention listing, free: for each backup on the node's manager-profile backup storage the pass reads the manifest (one small GET) and checks every artifact it names is in the listing at the recorded size, and that the manifest carries its envelope (FleetBackupRetention::check_shelf; compare_manifest is the pure rule). What it finds is stamped on mgn_backup_shelf_problem — one line in the pass's words, empty when every backup is whole — and the health check turns anything else into "A backup in backup storage is incomplete" on the node's card. It catches a partial upload, an object deleted out from under retention, and a manifest rewritten after its artifacts were pruned. A manifest the pass could not read (a transport error, an HTTP status) is a fact about the pass, not backup storage: it is named in the pass's report and the stamp is left as it was, so a network blip is neither shown as an incomplete backup nor clears a real one found last time. The check runs on every retention listing, which is once per backup cycle for each node, not on every tick. Every offloaded file an index it read marks stored and the listing lacks goes to that node's run in the same pass as objects_missing (to an agent at 1.83.0 or newer, a page that fits in the job), and the run stores those files again from the site's copy (Backups § Objects backup storage lost). The object half reads every restore point's index in each chain it holds, not only the newest, and names the restore points that cannot bring a file back by their times (compare_chain_indexes); earlier restore points that name a lost location stay reported until retention removes their chain.
• Opened and read is dispatched as a verify_backup job (level 2, of the newest backup in backup storage) when FleetBackupPolicy::is_verify_due() says so: the policy's verify_every_days is above zero (fleet default server_manager_fleet_backup_verify_every_days, 7; 0 means never, stored as a decision), the node has a successful backup from here, and either no verify has ever been attempted, or the last attempt is older than the interval and a newer backup exists. The last attempt is the later of the node's verify stamp and the creation of its newest verify_backup job, whatever became of that job (FleetBackupPolicy::last_verify_attempt): a verify that fails on the node comes back as a failed job, and a failed job is never folded into the node's columns, so the stamp alone would read "never verified" and re-dispatch the whole download every tick. A verify takes its machine's turn the way a backup does and is never dispatched beside a running backup, Prepare or verify of the same node; a backup due on the same tick waits for it. A failed verify is never retried automatically. Level 3 is never scheduled: it is a person's choice on the Backups tab.

A backup problem is an incident (see Incidents), judged exactly as the node's Backups tab judges it (NodeMonitorHealth::fleet_backup_health(), whose result carries a kind): the last run failed (plane:backup_failed, critical, with the reason the node gave and its run); backups are not happening (plane:backups_stopped, critical: no verified recovery key on the node, never backed up, not landing in storage, or the last success older than the schedule allows; a node in its first seven days with no recovery key yet opens none, since only its own administrator can set one up, while its Backups tab says so from the first pass); or a backup was taken but is not proven whole (plane:backup_unverified, a warning: a suspiciously small run, an incomplete stored backup, or verification failed, stale or never done). A node the scheduler would not back up, or whose fleet backups somebody switched off, has none. Each clears when the judgement does.

The node's word is cross-checked against the bucket. The scheduler lists each node's backup storage with this management node's own credential on the first pass after the node reports a successful run (FleetBackupRun::witness_landing, due while the claimed run started after the last listing; a failed listing is retried next pass), and the retention pass lists it again before every run. Each listing stamps what it saw — when backup storage was listed and the newest object write on it — onto mgn_backup_shelf_checked_time and mgn_backup_shelf_newest_time. The health check compares that against the node's claimed last run: backup storage listed after a claimed success that holds nothing written since raises "Backups are not landing". Backup storage is the one witness a compromised or misconfigured node cannot talk into its story — everything else in the health picture is the node reporting on itself.

A node with fleet backups switched off produces nothing either — that was somebody's decision.

Whether a backup is verified restorable is the next question the card answers, from mgn_backup_verify_* (stamped by process_verify_backup from the job's VERIFY_* lines, and adopted from the node's own status report when the node verified itself and that is newer — adopt_reported_verify). NodeMonitorHealth::verify_state() gives four answers: a backup storage problem (above) is a problem; a verify that failed is a problem, in the node's own words, with the note that nothing is retried automatically; a pass older than a day past the node's verify interval (FleetBackupPolicy::verify_alarm_days(), 8 days at the weekly default) is stale, a problem; and never verified is information on a healthy card ("Not yet verified restorable") for the same number of days after the first backup from here, after which it is "Backups never verified restorable". A verify that was skipped (not enough disk — "needs N free, has M" — or a busy machine) records only its reason, beside the last real result.

Which key each node holds, and whether it can be backed up

set_recovery_key.php --report is asked during check_status, and the answer lands on mgn_backup_recovery_fpr and backup_recovery_state. It prints one machine-readable line, RECOVERY_KEY=already|none|invalid, and the fleet table on the Targets page reads the columns rather than reaching out to every node on page load.

That state decides whether the node can be backed up at all, by anyone — the Targets page lists it as fleet coverage, and RecoveryKeyFleet::has_own_key() is the one predicate every surface asks. Whose key it is is not compared against this management node's: a node holding a key this machine has never seen is a node whose operator holds their own recovery key, which is the intended arrangement.

It is reported and never written. There is no job type that can write it, and set_recovery_key.php refuses --public outright so a stale management node finds out rather than succeeding. A node's key is set up on that node's own Backups page, with the possession ceremony that makes it trustworthy — the page generates a keypair in the browser and runs the challenge in one pass.

Disaster recovery

A lost node is rebuilt on a new machine from its offsite backups as a copy from backups on its Copy tab (A copy run from backups, above): the new machine reads the newest chain at its storage provider itself, its owner opens the chain's key with the recovery key on the new machine's own page, and the copy restores it, takes the chain's secret_box_key so every sealed value opens, and switches over under the node's own id.

By hand, with no management node:

1. Fetch the archive and its envelope ({archive}.keys.json) from the bucket.
2. Recover the archive key on a machine holding the recovery private key: php backup_envelope.php open --sidecar {archive}.keys.json --private /path/to/recovery.key --key-out /tmp/k
3. Install a fresh site on the new machine, then restore onto it with restore_project.sh --key-file /tmp/k (or `restore_chain.sh --key-file /tmp/k --adopt-secret-key for a chain, which takes the chain's secret_box_key` so sealed values open). Without that key the new machine keeps its own, so every sealed value from the old one is dead and is entered again (Sealed Secrets).

{
    "steps": [
        {"type": "ssh", "label": "Prepare the workspace", "cmd": "mkdir -p ..."},
        {"type": "ssh", "label": "Run the operation", "cmd": "...", "continue_on_error": true}
    ]
}

A job's steps form two phases. Steps without the teardown flag are the main phase: they run in order, and a hard failure (no continue_on_error) stops the phase and determines the job's outcome. Steps flagged "teardown": true are the teardown phase: they run on every exit path — success, mid-job failure, or none-of-the-main-steps-ran — so the scratch files a job creates (dumps, staged archives, unpacked installers) are removed even when the job aborts on a shared production host.

Teardown semantics:

• Teardown never changes the outcome. A failed job stays failed with the original failing step in mjb_error_message; a teardown step erroring is logged under the === Teardown === output header and ignored.
• Teardown runs before the terminal status is written. The job stays running while teardown executes, so the per-node concurrency lock holds (no re-run can race the deletions) and the job detail view keeps streaming.
• Progress counts main steps only. mjb_total_steps excludes teardown steps and teardown output never advances mjb_current_step.
• Stale-job replay. Jobs force-failed at agent startup (left running by a crash or restart) get their teardown steps replayed from mjb_commands — safe because every teardown command is an idempotent rm on a per-job path.
• Placement. Builders put teardown steps at the tail of the array, after every main step, and keep continue_on_error on them. An agent that ignores the flag runs the array sequentially, so tail placement makes the steps plain trailing cleanup there — correct, just not failure-proof.

// plugins/server_manager/includes/JobCommandBuilder.php
public static function build_restart_apache($node) {
    return [
        ['type' => 'ssh', 'label' => 'Restart Apache',
         'cmd' => 'systemctl restart apache2'],
        ['type' => 'ssh', 'label' => 'Verify Apache status',
         'cmd' => 'systemctl is-active apache2'],
    ];
}

$steps = JobCommandBuilder::build_restart_apache($node);
$job = ManagementJob::createJob($node->key, 'restart_apache', $steps, null, $session->get_user_id());
header('Location: /admin/server_manager/job_detail?job_id=' . $job->key);

$job = ManagementJob::createJob(
    $node_id,               // target node ID (or null for local)
    'backup_run',           // job type label
    $steps,                 // array of step dicts from JobCommandBuilder
    ['profile' => 'manager'], // parameters (stored for reference/re-run)
    $session->get_user_id() // who triggered it
);

A lightweight per-node uptime check runs on each scheduled-task tick (~15 min). It updates live state on mgn_managed_nodes. A site that goes down is an incident (plane:site_down, see Incidents): the reconciler opens it from these columns, tells the superadmins once, and clears it when the site answers again.

Augmented mgn_managed_nodes fields:

• mgn_uptime_enabled (bool, default true) — per-node on/off
• mgn_uptime_check_type (varchar, default 'http_status') — which check method to use (see below). http_status is the default because it concludes up/down for any node with a site URL and needs no setup; api is an opt-in that requires API keys provisioned on the node.
• mgn_uptime_last_status (varchar) — 'up' / 'down' / null (never checked)
• mgn_uptime_consecutive_failures (int) — streak counter for threshold logic
• mgn_uptime_down_since (timestamp) — when current outage started, null when up
• mgn_uptime_down_reason (varchar) — what the newest failing probe saw while down (the incident's evidence), null when up
• mgn_cert_expiry_ts (timestamp) — observed notAfter of the served TLS cert (see Certificate monitoring)
• mgn_cert_problem (jsonb) — the served certificate's current problem (reason, title, not_after, detail), or null; the certificate incident's evidence

Per-node IP pinning. http_status checks pin the request to the node's own mgn_host IP (CURLOPT_RESOLVE, SNI/Host preserved) when mgn_host is an IP literal and appears in the site hostname's public A records (DnsResolver::getA()) — the same directly-exposed guard check_cert_expiry() uses. A node behind a shared or round-robin hostname (e.g. two DNS servers sharing dns.scrolldaddy.app via dual A records) is therefore checked as itself, not whichever A record DNS happens to return — otherwise a single dead node hides behind its live partner. The guard matters: pinning a Cloudflare-fronted node to its origin IP would bypass the edge and hit the Apache default-vhost fallback cert, failing SNI/cert validation and reporting a false down — so nodes that aren't directly exposed are checked unpinned, through the public hostname, same as before pinning existed.

Check types (extensible via a single dispatch switch in RunNodeUptimeChecks::run_check()):

It reads the served certificate over the wire (stream_socket_client on ssl://mgn_host:443, capture_peer_cert, SNI = the site hostname), so it sees whatever the cert manager actually serves — Caddy, certbot, anything. Validity is deliberately not* verified, so the notAfter of an already-expired or near-expiry cert is still readable.

Fronted nodes are watched the same as direct ones. The probe is pinned to mgn_host, so a name proxied through Cloudflare is answered by the origin, not the edge. The edge renews its own certificate; the origin behind it holds another that expires on its own schedule, and on Cloudflare Full (not Strict) an expired origin certificate is accepted and invisible — right up until Strict is enabled and it is an outage. One test decides whose certificate it is: the served cert's CN/SANs must cover the hostname (cert_covers_host()). A certificate for some other name is not this node's to date, but it is reported (reason 4 below).

On a monitored node it stores mgn_cert_expiry_ts and, in mgn_cert_problem, the first of five reasons (cert_alert_verdict(), www_gap(), uncovered_problem(), placeholder_problem()), or null when there is none:

1. Renewal overdue. A standard ACME client replaces a certificate at two thirds of its life (renewal_due_ts()); when that date is more than a day past and the certificate on the wire is still the one that was due, renewal on the node is failing. No stored fingerprint is needed: the due date is computed from the served certificate's own dates, so a replacement carries a fresh notBefore and its own due date lies ahead. For a 90-day certificate this fires around day 61, nine days before the expiry threshold would.
2. Expiry approaching. Days remaining under server_manager_cert_expiry_warn_days (default 21) — the safety net for a certificate whose issue date cannot be read.
3. www uncovered. When www.<hostname> resolves, the origin is probed again with www. as SNI; a certificate that does not cover it is its own reason, because a Strict flip would take the www address dark while the apex looks healthy. A www that does not resolve is not mentioned.
4. Origin uncovered. The origin answers TLS but with a certificate for other names — a shared fallback vhost on the host. Under Full the site serves; under Full (Strict) it goes dark. The problem names the presented names and the wanted one and the tool that fixes it (issue_origin_cert.sh). Nothing is stored for a foreign certificate: its expiry is not this node's.
5. Origin on its placeholder. The origin answers with a self-signed certificate (is_self_signed(): its issuer is its subject), the placeholder a site is given until its real certificate is issued. It names the site, so it covers the name, but nothing renews it and Full (Strict) or a visitor reaching the origin directly refuses it. The problem carries when it was first seen (since, kept across checks by placeholder_since()) and the fix: setup_ssl.sh on the site's own server, or the agent bundle's setup_ssl.sh on its host for a site in a container. Its ten-year expiry is not stored.

This is distinct from mgn_ssl_state / the SSL tile, which track certbot provisioning status (does an LE cert exist on disk) — a different question from "is the served cert about to expire," and largely a different set of nodes. The two are orthogonal.

Safety Constraints

1. No safety copy before a restore -- restore_database and restore_project replace the node's database (and, for restore_project, its project tree) with the archive, and nothing is saved of what they replace. The operator approves the destructive job on the node itself, and the approval states that anything written since the archive was taken is gone.

1. Database restores replace -- A database restore leaves the target equal to the snapshot. Every restore verifies the archive with gunzip -t before anything is destroyed, drops and recreates the public schema so target-only objects are removed too, and loads with psql -v ON_ERROR_STOP=1 so the first load error fails the job instead of completing a partial restore. Dumps are plain pg_dump snapshots -- the restore step owns the replacement guarantee, so it holds for any file it is fed. Backup files restore onto the site that made them, where the database role matches.

1. Per-node concurrency lock -- The agent skips jobs if another job is already running on the same node, preventing conflicts.

1. Stale job recovery -- On agent startup, any orphaned running jobs are marked failed with a descriptive message.

1. Step timeout -- 30-minute default per step, overridable. On timeout, the SSH session is killed.

1. Single-threaded agent -- One job at a time. Queued jobs run sequentially.

1. Remote credentials at runtime -- Database credentials for backup/copy/restore are extracted from each node's Globalvars_site.php at execution time, never stored on the management node.

API Actions

The dashboard's page JS calls these POST /api/v1/action/server_manager/{name} actions with the browser-session credential (superadmin only, floor 10) and reads the response envelope's data. The full set: probe_api, job_status, backup_actions, refresh_node_status.

server_manager/job_status

Polled by the job detail page for live output.

Parameters:

• job_id (int) -- job to query
• output_offset (int) -- character position; only new output since this offset is returned

{
    "success": true,
    "status": "running",
    "new_output": "=== [Step 2/5] Check memory ===\n...",
    "output_offset": 1234,
    "current_step": 2,
    "total_steps": 5,
    "error_message": null
}

The UI polls every 2 seconds while a job is running and stops when status is completed or failed.

server_manager/backup_actions

Used by the backup browser on the Backups tab.

Troubleshooting

Agent shows Offline on dashboard

• Check the agent is running: sudo systemctl status joinery-agent
• Check logs: journalctl -u joinery-agent -f
• Verify DB credentials in /etc/joinery-agent/joinery-agent.env match those in Globalvars_site.php

• Agent is not running or can't connect to the database
• Another job is running on the same node (per-node lock)

• Verify SSH key path on the node record matches an actual key file
• Test manually: ssh -i /path/to/key root@host "echo ok"
• For container nodes, verify the container name is correct

• The agent crashed or was restarted mid-job. Check journalctl for the crash cause.
• The partially-completed job should be inspected manually. Use Re-run to retry.

File Reference

Plugin (plugins/server_manager/)

Go Agent (/home/user1/joinery-agent/)