Server Manager

The Server Manager plugin provides a web UI for managing remote Joinery production servers. Operations include status checks, backups, database copies, and applying updates -- all from the admin interface at /admin/server_manager.

The system has two components:

• PHP plugin (plugins/server_manager/) -- admin UI, job creation, command generation
• Go agent (/home/user1/joinery-agent/) -- runs a management node's own job queue, and on a managed node takes work from its plane over the agent channel

Quick Start

1. Install the plugin

The plugin is already in the plugins/ directory. From the admin panel:

1. Go to /admin/admin_plugins
2. Click Actions on "Server Manager" and choose Install
3. Click Actions again and choose Activate

2. Install and start the Go agent

Release channel (how the agent normally arrives and stays current)

The agent ships inside the platform release. Publishing an upgrade bundles a signed agent artifact into public_html/agent_dist/ — core, not this plugin, because every node must receive it and no plugin may arrive as a side effect of a core upgrade (this one is commercial and entitlement-gated). The plugin builds and signs the artifact; the core archive delivers it:

• manifest.json — agent version plus, per architecture, the artifact filename, its sha256, and an Ed25519 signature over the raw binary
• joinery-agent-linux-amd64.gz / joinery-agent-linux-arm64.gz — the binaries
• joinery-agent.service — the systemd unit

Bundling is the first thing a publish does, before the VERSION file, the archives or the release row, because its outcome decides whether the release happens at all:

• No agent source on this box — the existing artifact carries forward unchanged and the publish proceeds. Publishing never depends on a Go toolchain being present.
• Source version matches the bundle — agent_dist is left byte-identical. It sits outside every plugin tree, so a rebuild changes no plugin's tree hash and bumps no plugin's version.
• Source is newer and the rebuild succeeds — the fresh artifact is bundled, and because this happens before plugin archives are built it is captured in the server_manager archive and its tree hash.
• Source is newer and the rebuild fails — the publish is refused. The build error is printed, and the VERSION file, archives and release row are all left untouched. Shipping here would mean releasing an agent the publisher already knows is out of date, and the resulting fleet has no way to tell.

First install is handled by the core installer maintenance_scripts/install_tools/install_agent.sh, which runs at every root moment — site install, code upgrade, container start, and the node-detail Run Plugin Installers action. It installs the bundled binary, writes the env file with the right JOINERY_CONFIG, and sets up systemd or cron supervision automatically.

The installer is core rather than a plugin's, and runs on every Joinery instance: the agent does a machine's own backups, upgrades and health checks, and only a management node has server_manager turned on. The artifact stays in this plugin's tree because this plugin builds and signs it, and it reaches every node regardless — the plugin is included_in_publish and receives_upgrades, both independent of whether it is active there.

The binary lands on every deployment. Installing is not running: the artifact is converged at each root moment regardless of the switch, so a machine that is switched on later starts a service that is already there rather than fetching, decompressing and verifying one at that moment.

Whether it runs is one setting, agent_enabled, which ships off and is read fresh at each root moment: on starts the agent and sets up its supervision, off stops it and takes the supervision away — the cron keepalive included, or stopping would last a minute. The agent's identity survives an off, so turning it back on resumes the same pairing. A database the installer cannot reach leaves the machine untouched rather than being read as off.

Three ways to set it, all writing the same setting:

• the machine's own Admin → System → Management Node page, which also says what is still needed for it to take effect
• php utils/agent_control.php --on (also --off, --join=URL, --leave, --status) on the machine
• from a management node, the node detail Agent Channel panel's Turn on the agent over SSH, which switches it on, runs the installer, and has the node ask to join — the fleet path, available only while SSH is, and retired with it at the Phase 3 cutover
• install.sh --enable-agent at install time. A site this management node provisions passes it automatically, so a node it builds comes up running its agent — the one case where whether the machine should run one is already answered

A web request has no root, so flipping the setting does not itself start the agent — the next container start or upgrade does, or sudo bash {site root}/maintenance_scripts/install_tools/install_agent.sh {sitename} does it immediately. The Management Node page says so, and prints that command when the switch is on and no binary has reached the machine yet.

Every later version change is handled by the agent itself. Between jobs, the agent compares its own version with the bundled manifest. When they differ, it decompresses the artifact, checks the sha256, verifies the Ed25519 signature against the public key embedded in its binary, keeps the current binary as .bak, renames the new one into place, and exits cleanly for its supervisor to restart. The signature check is the security boundary: the site tree is writable by the web user while the agent runs as root, so the agent never installs anything the publisher did not sign. An artifact that fails verification is refused, logged under a === Self-update === header, and not retried until the manifest changes.

If the new binary fails to initialise (config, DB, or schema), it restores the .bak over itself and records the bad version in a .rejected marker — the supervisor restarts the previous working agent, and that version is never reinstalled; the next release supersedes the rejection. On the first fully healthy start after an update, the .bak and any stale marker are removed.

The dashboard's Agent Status bar surfaces all of this from the heartbeat row (ahb_bundled_version, ahb_update_state): a pending update, a refused (verification-failed) artifact, a rolled-back version, or an agent built without an update key.

Manual install (bootstrap fallback)

For a management node that has no bundled artifact yet, build and install by hand:

cd /home/user1/joinery-agent
PUBKEY=$(cat /var/www/html/joinerytest/config/agent_signing_key.pub) make release
sudo bash joinery-agent-installer.sh --verbose [--config /path/to/Globalvars_site.php]

make release compiles the binary (passing PUBKEY bakes in the update-verification key so the manual build can still self-update later) and packages it into joinery-agent-installer.sh, a self-extracting script that handles both fresh installs and upgrades with automatic rollback if the new version fails to start.

The installer detects the host's supervision capability:

• systemd hosts: installs /etc/systemd/system/joinery-agent.service; start with systemctl start joinery-agent.
• No systemd (Docker containers, minimal hosts): installs /usr/local/bin/joinery-agent-supervise plus /etc/cron.d/joinery-agent (@reboot + a once-a-minute keepalive) and starts the agent immediately. Logs go to /var/log/joinery-agent.log.

• /usr/local/bin/joinery-agent — the binary
• /etc/joinery-agent/joinery-agent.env — configuration (from example, first install only)

Every management node needs a live agent. All jobs (install_node, provision_ssl, backups, upgrades) sit pending until an agent polling that site's own database claims them. The Server Manager → Provisioning page shows an agent heartbeat badge as requirement #1.

Configure (usually not needed)

The agent reads database credentials directly from Globalvars_site.php — no manual configuration required on a standard Joinery install.

The default config path is /var/www/html/joinerytest/config/Globalvars_site.php. If your install is at a different path, set it in the env file:

sudo nano /etc/joinery-agent/joinery-agent.env
# Set: JOINERY_CONFIG=/var/www/html/mysite/config/Globalvars_site.php

Other optional settings in the env file:

Start

sudo systemctl start joinery-agent
sudo systemctl status joinery-agent

The dashboard at /admin/server_manager should now show Agent Status: Online.

If anything is wrong, the agent logs to the systemd journal (systemd hosts) or /var/log/joinery-agent.log (cron-supervised hosts):

journalctl -u joinery-agent -f      # systemd
tail -f /var/log/joinery-agent.log  # cron supervision

Common startup errors are self-explanatory — missing DB_NAME, wrong password, or plugin tables not installed. Each error message tells you exactly what to fix.

Upgrade

Agents upgrade themselves from the bundled artifact after each platform release lands (see Release channel above). The manual-install path also accepts upgrades: re-running the generated installer stops the service, swaps the binary, restarts, and rolls back automatically if the new version fails to start.

3. Add managed nodes

Go to /admin/server_manager/node_add (or click Add Node on the dashboard). There are two ways to add nodes:

Auto-detect (recommended)

The Auto-Detect Joinery Servers panel scans a remote host for Joinery instances automatically. Enter:

1. SSH Host -- the server IP (e.g., 23.239.11.53)
2. SSH Key Path -- path to the private key on the management node (defaults to /home/user1/.ssh/id_ed25519_claude)
3. Click Detect

Detected instances appear as cards with Add This Node buttons. Clicking one auto-fills the entire form below -- just click Add Node to save.

Auto-detect requires the Go agent to be running (it executes the SSH commands, not PHP).

Manual

Fill in the form fields directly:

Admin Pages

All pages are at /admin/server_manager/... and require permission level 10 (superadmin).

The UI is organized around a dashboard + node detail pattern. The dashboard shows the fleet overview; clicking a node opens a tabbed detail page with all operations for that node.

Node Detail Tabs

The node detail page (/admin/server_manager/node_detail?mgn_id=N&tab=...) has six tabs:

Running a command on a node

There is no way to. A management node holds no mechanism for running an instruction it composed at runtime on a managed node — no Console tab, no node_exec.php, no run_command job (decision A1).

What a node accepts instead is a primitive: a name it looks up in a vocabulary compiled into its own agent, with parameters validated against a declared shape. A job names an operation; it never carries a command. A node refuses anything outside that vocabulary whatever this plane asks, so what the fleet can be made to do is bounded by what was built and signed, not by what someone can compose.

Work with no primitive yet is done over the operator's own SSH key. That is deliberate and it costs something real: those sessions are not recorded, where a console run produced a job row with its command, operator, output and exit status. The audit trail was given up knowingly, because a recorded path that can run anything is still a path that can run anything, and a compromised plane would have used it. sshd stays on across the current fleet (A11), so this is the working route until the vocabulary covers the operation.

Dashboard Features

The dashboard shows:

• Agent Status -- online/offline indicator with version and last heartbeat time
• Managed Nodes -- cards with health-based status dots (green=healthy, yellow=warning, red=problem, gray=no data), key metrics, and action buttons
• Publish Upgrade -- build upgrade archives from management node source code (node-independent)
• Recent Jobs -- latest 20 jobs across all nodes

• Red: Last check failed, disk > 90%, or PostgreSQL not accepting connections
• Yellow: Disk > 80% or load average > 5
• Green: All metrics healthy
• Gray: Never checked or no data

The agent channel

A node's agent polls its management node over an outbound HTTPS connection, takes one job at a time, and posts the result back. Nothing has to reach in: a node behind NAT or Cloudflare works the same as one with a public address, and the poll itself is how the plane knows the node is alive.

A job names an operation; it never carries a command. The payload is {primitive, params} — a name the agent looks up in a vocabulary compiled into its own binary, plus parameters validated against the bounds that primitive declares. A name the agent does not have, a parameter it did not declare, or an operation class the node does not accept is refused on the node, with the reason recorded and reported back, whatever the plane asked for.

Primitives are grouped into three classes, and a node's acceptance policy is set per class:

The policy lives at /etc/joinery-agent/policy.json, root-owned, outside the web tree. A missing file means the shipped fleet-wide policy above. A file that is not root-owned, or is writable by group or other, is refused outright and the node accepts nothing until a human fixes it — trusting a file the web user could have written would defeat the only thing the file is for.

Connecting a node (the join)

Enrollment is node-initiated and shares no secret. On the node, Admin → System → Management Node: the site's admin enters the management node's URL — just an address. The node's root agent generates an Ed25519 keypair, keeps the private half at /etc/joinery-agent/node_identity.json (mode 0600, root-owned), and sends a join request carrying only the public half and a claimed name. The node's page then shows the key's short fingerprint and waits.

On the management node, the request appears on the node's Detail → API Keys tab: claimed name, source address, and the same fingerprint. Approving is superadmin-only and should happen only if the fingerprint matches the node's own page exactly — the name and address are claims anyone could make; the fingerprint comparison is the entire security of the introduction. Approval binds the public key to the node record; the agent picks it up on its next check and both panels flip to Connected.

The management node stores only the public half, so there is no credential on it that could act as the node — and enrollment adds nothing to steal: no token, no code, nothing copied by a human. A wrong approval is visible (the tab stamps the connection time, agent version, and last poll) and severable with the Disconnect button, which forgets the key and drops the node back to API/SSH routing.

Requests are rate-limited, expire after an hour (the agent renews its own while it waits), and are capped in number. A rejected request retires its keypair — the agent introduces itself with a fresh key on the next ask.

The web tier's only involvement on the node is a handoff through three managed settings: agent_join_request (the URL the admin asked for), agent_join_state (the agent's progress, which the page renders), and agent_leave_request (the admin asked to disconnect). None ever holds a credential.

Routing

A connected agent is routed to — approving the join is the routing decision, and there is no further switch. Any operation with a primitive implementation runs on the node's own agent; everything else routes over the API and SSH. Disconnecting the agent is what returns all of a node's work to API/SSH.

Either side can end the pairing, and neither needs the other's cooperation. This plane's Disconnect button forgets the node's key. The node's own Management Node page has a Disconnect too: its agent finishes any running job, sends one signed goodbye to /api/v1/agent/leave (so this plane forgets the key immediately), deletes its identity, and returns to serving only local work — and it leaves even when the goodbye cannot be delivered, in which case this plane just sees the agent go silent until someone disconnects the node here as well. Both endings run through AgentChannelEndpoint::forgetAgent(), so they cannot drift apart.

Operations cross one at a time. An operation has crossed when JobCommandBuilder has a build_<op>_primitive method for it; transports_for() then lists primitive ahead of api and ssh, and ManagementJob::createFromBuild() stores the right shape without any caller knowing which transport ran.

A primitive is routed to a node only when that node says it has it. Every claim carries the agent's own list of the primitives its binary compiled in, and the plane stores it (mgn_agent_primitives). Routing consults that list: an operation missing from it is dispatched over API or SSH instead, whatever version the node reports. The node's own account is the only one that is not a guess — a version number says which release a machine runs, and only the machine says what that release compiled into it.

An agent old enough not to send a list leaves the column empty, and those nodes are answered by JobCommandBuilder::PRIMITIVE_MIN_AGENT_VERSION: a per-operation floor naming the agent version that introduced the primitive. A node below the floor, or with no known version, routes away from the primitive — dispatching to a vocabulary that cannot be confirmed trades a working transport for a certain refusal.

Upgrade the management node's own agent before connecting any node. An agent from before this channel existed does not know to leave primitive jobs alone, and would claim one out of its local queue. Such a job carries a step type no released executor recognises, so that agent fails it and says exactly why rather than marking it complete having done nothing — but the job still has to be re-run.

Reading refusals

A refused job is a terminal failure like any other, and reads as one everywhere a job status is shown — the message says Refused by the node: and then the node's own reason. The outcome the node actually reported is also recorded on its own (mjb_agent_outcome: completed, failed or refused), so a refusal can be counted rather than found by matching the text of an error message. ManagementJob::refusalCountForNode() is that count, windowed.

This matters more as the vocabulary grows. A node refusing work is a node whose plane is asking for something it should not be, or whose policy has been tightened without the plane noticing — either way it is the number, not the prose, that an alert reads.

A job this plane gives up on after repeated lost claims records no node outcome. The node never reported one, and inventing a verdict for it would make the refusal count untrustworthy in exactly the situation where it is being consulted.

When a claim does not come back

An agent claims a job and then reports. If it never reports — it crashed, the box rebooted, the network went — the job would otherwise sit in running holding that node's concurrency lock. A claim older than 15 minutes is returned to the queue with a note in the job output saying so, on every poll and on every scheduled uptime pass. After three such claims the job fails instead, naming the node: a job that kills three agents will not succeed on the fourth.

Endpoints

POST /api/v1/agent/join, /join_status, /claim, /result, /leave, /quiet, /artifact. Join and join_status are unauthenticated by nature — until approval there is no identity to authenticate — and grant nothing; they are rate-limited, validated, and bounded in number. Claim and result are signed with the node's key; the plane verifies against the public half it holds, and selects jobs by the identity the signature proves, never by anything in the request body. Requests are schema-validated, size-capped, and refused for a clock more than five minutes from the plane's. These are a separate route family from /api/v1/management/*, which runs the other way — the plane calling in to a node's web tier — and stays status-only.

/artifact is the one response that is bytes rather than a JSON envelope, and it is described under Machines with no site. Its body fetches are metered on their own bucket (api_agent_artifact_rate_limit_requests, per address per window) and each is recorded, because they are the expensive ones; the small manifest fetches ride the general agent-channel limit.

Machines with no site

Some machines a management node manages host no Joinery site at all — a mail relay, a Docker host. They run the same agent in a machine posture: no site root, no local database, no admin page, and no platform release ever delivered to them. Two things follow, and both are served by the same endpoint.

Keeping the agent current

A machine with a site tree finds its next binary in public_html/agent_dist, put there by its own upgrade, and updates without asking anyone. A machine with no site has no such directory, so it fetches from its management node instead: a signed request to /api/v1/agent/artifact for the dist manifest, and — when the version differs from what it is running — for the binary for its architecture.

Nothing about verification moves. The agent decompresses, checks the sha256, and verifies an Ed25519 signature against the public key compiled into its own binary at build time, exactly as it does for a locally delivered artifact. The management node does not hold the release key and cannot sign an agent, so a plane serving a hostile binary produces a refusal and a recorded rejection. The endpoint is a delivery route for bytes that were always verified on arrival.

The request names a kind from a closed set and, for a binary, an architecture matched against a pattern. It never names a file: the plane resolves what to send out of its own manifest, so nothing a node sends is read as a path.

The support bundle

A script-invoking primitive verifies its script against the signed release manifest before running it as root. On a machine with no site there is no release manifest, so there is nothing to verify against and no script primitive can run at all — which would leave the machine's whole vocabulary in embedded Go.

The support bundle closes that. A publish builds public_html/agent_dist/support_bundle.tar.gz: a small tree carrying the scripts those machines' primitives invoke, at site-root-relative paths, with its own RELEASE_MANIFEST and .sig signed by the release key. A siteless machine fetches it over the same artifact endpoint, verifies the signature against its baked-in key, checks every listed file's hash and that the tree holds nothing the manifest does not list, then unpacks it root-owned to /opt/joinery-agent/tree. Script primitives resolve against that tree when there is no site root; a machine with a site root uses the site root, and a machine with neither refuses as it always has.

The bundle's contents are a deliberate list in SupportBundlePublisher, not a directory sweep — every entry is a script some primitive names, and adding one is a visible decision. A script that sources another needs that sibling in the list at the same relative path. Binaries the scripts invoke ship under bin/ with one file per architecture (bin/<tool>-linux-amd64, bin/<tool>-linux-arm64) and the script selecting on uname -m: one bundle carries both, so no machine can end up holding one built for the wrong architecture.

Its version is the hash of its own manifest body, so a publish that changes no bundled script leaves the bundle byte-identical and no machine downloads anything. The plane also advertises the tarball's sha256, which an agent uses only to skip a transfer it already has — what makes the tree runnable is the signature inside it.

Each machine reports the bundle version it holds on every claim, stored as mgn_agent_bundle_version. On a machine nobody logs into, that column is the evidence the bundle arrived.

Enrolling and switching one on

There is no admin page on these machines, so the ceremony is the same one reached from the command line:

joinery-agent join --management-node=https://plane.example.com
joinery-agent status
joinery-agent enable | disable
joinery-agent leave

join generates the keypair, sends only the public half, and prints the fingerprint to compare against the pending request on the plane — the same comparison, and the same approval, as a node enrolling from its own admin page. The run switch is the marker file /etc/joinery-agent/enabled, which enable and disable write directly: on a machine with no settings table, the marker is the switch rather than a projection of one.

Install with install_agent.sh --siteless, which is explicit and never inferred — a missing site config keeps meaning "not my machine, exit 0" for everything else. --dist-dir=DIR names where the first artifact is, since no release delivered one.

Job Types

Note on bare-metal nodes with user1 SSH: When a bare-metal install completes, install.sh disables root SSH and the node's mgn_ssh_user is automatically updated to user1. Subsequent jobs run as user1 with NOPASSWD sudo. All backup/restore commands that need root-level paths (e.g. /backups/) use sudo automatically.

One-Click Node Install

Dashboard → Install New Node opens a form that provisions a Joinery site in a single click. The Target Host dropdown offers three kinds of target:

• A known host (or Other server with manual SSH details): the form creates the ManagedNode and dispatches the install_node job immediately.
• Create a new cloud instance: no server exists yet. The form records an admin-origin CustomerCloudProvision (connected cloud account, region, instance type, plus all install parameters) and the Provision Customer Cloud task births the instance, creates the node, and dispatches the install — see Customer-Cloud Fulfillment. The instance is created in, and billed to, the selected connected account; Linode grants expire after two hours, so connect (or re-connect) shortly before submitting. Cloud targets always take a fresh source backup in From-Backup mode. In-flight provisions appear in a banner at the top of the dashboard.

Two install types:

• Fresh: empty Joinery site with default schema. Admin picks the domain. The admin login is [email protected] with a password generated for that site alone — there is no shared default. Like the generated Postgres password, it stays on the node rather than in the management node: read it at /var/www/html/{sitename}/config/admin_credentials.txt (root only), or set a new one with maintenance_scripts/sysadmin_tools/reset_admin_password.php. usr_force_password_change=true, so the first sign-in forces a new password.
• From Backup: fresh install + restore of a source node's DB and project files, then reconciliation to the new node — its own domain (the node's recorded URL), its own deployment shape, its own paths. Use source admin credentials to log in; cut DNS over when ready and the certificate is issued on its own.

From-Backup restores files by extracting the source archive with both of its leading path components stripped, taking only the project_files/ subtree — backup_project.sh writes archives as `{backup_name}/project_files/{public_html, uploads,config,...} with the archive's own metadata (apache_config/`, backup_info.txt, the .sql dump) as siblings. The target keeps its own Globalvars_site.php (it holds this machine's database password and secret_box_key) and mints its own backup_site_key rather than inheriting the source's identity as a backup recipient. A verification step then requires every regular file the archive carries to exist at the site root and fails the job otherwise, because a clone whose files did not land still serves pages: the fresh install ran first and the database restore succeeded, so the only symptom is uploaded files missing from where the restored database says they are.

The mgn_install_state column tracks the lifecycle: installing → NULL (success) or install_failed (failure). On failure, the node detail page surfaces a Retry Install button; the target must be cleaned manually (e.g. rm -rf /var/www/html/SITENAME) before retry because install.sh refuses to overwrite an existing site. Postgres passwords are auto-generated and stored in the target's Globalvars_site.php — Server Manager does not capture or display them.

Docker notes:

• The reverse proxy step (manage_domain.sh) is skipped when the domain is a bare IP address — a routable hostname is required for Apache ServerName-based virtual hosting. With an IP domain, the site is accessible directly on its mapped port.
• backup_project.sh requires rsync. The bare-metal and Docker install scripts install rsync as part of the essential packages (install.sh line ~948). Sites installed before this was added can install it manually with apt install rsync.
• After a Docker install, mgn_container_name is automatically recorded in the management node DB so future jobs correctly use docker exec to reach the site.

SSL Management

SSL State

Each node tracks its TLS certificate state in mgn_ssl_state:

Automatic Detection

check_status jobs include an SSH step that checks for a Let's Encrypt cert under /etc/letsencrypt/live/{domain}/. JobResultProcessor updates mgn_ssl_state and stores ssl_domain, ssl_expiry_raw, and ssl_expiry_ts in mgn_last_status_data. State transitions:

• CERT_FOUND → sets state to active (from any prior state)
• CERT_MISSING → clears state to null only if currently null or active; never overwrites pending or failed

Manual Provisioning

The Overview tab shows an SSL Setup card when mgn_ssl_state is not active, the node has a domain in its site URL, and mgn_cert_expiry_ts is empty. The last condition excludes directly-exposed, self-renewed nodes (see Certificate Expiry Monitoring) — their cert lifecycle is owned by an external renewer (e.g. Caddy), and the card's certbot-based provisioning does not apply to them. The card:

1. Resolves the domain via DNS and shows whether it points to the node's host IP
2. Enables the Provision SSL button when DNS is ready (or when the host IP is not configured)
3. On submit: creates a provision_ssl job, sets mgn_ssl_state = 'pending', redirects to job detail

Cloudflare-proxied domains skip certbot (Cloudflare terminates TLS at its edge) but are gated on a routing probe: the job writes a one-time token to {webroot}/sm-ssl-probe.txt on the node, and the management node fetches /sm-ssl-probe.txt through the domain. The token is only fetchable because core serve.php routes that URL to views/sm_ssl_probe.php, which serves the file — a Joinery front controller never serves arbitrary webroot files, so a node whose code predates that route cannot pass the probe and needs an upgrade first. Only a match — proof that traffic for the domain actually lands on this node — patches the proxy's X-Forwarded-Proto and marks SSL active (JobResultProcessor additionally requires the CF_ROUTING_VERIFIED marker). A miss fails the job and the domain stays pending until the customer's DNS actually routes here.

Automated Provisioning (installs only)

For nodes installed via Install New Node, ProvisionPendingSsl (scheduled hourly) watches for nodes with mgn_ssl_state = 'pending', checks DNS, and kicks off provision_ssl jobs automatically. After ~16 hours of failed attempts it flips state to failed — except a Cloudflare domain still waiting on its DNS cutover (CF_ROUTING_UNVERIFIED), which never flips: a cutover the customer has not made is not a fault, and can legitimately take days. Instead the routing wait is paced — hourly for the first ROUTING_FAST_ATTEMPTS tries, then one try every ROUTING_SLOW_GAP (six hours) — and entering the slow lane emails the operator once (recipient chain: server_manager_provisioning_admin_alert_email → webmaster_email → first superadmin; the sent marker is stamped into a job row's parameters as routing_alert_sent). The slow lane and the alert apply only while the domain still resolves to Cloudflare: once it repoints, the next attempt is due within the hour, and the 16-hour give-up window opens fresh at the first non-routing failure — time spent parked at Cloudflare never burns it. Manual provisioning via the Setup card is the fallback.

Hosting Provisioning

Paid hosting orders on getjoinery become installed, SSL'd Joinery sites with no human touch. The Poll Hosting Orders scheduled task polls the getjoinery API each cron tick for paid orders carrying an answer to the configured domain Question, and fulfills each one in the mode the product declares in pro_fulfillment_provider:

• Shared host (the default, any other value): the pipeline picks the least-loaded provisioning-enabled ManagedHost, assigns the next Docker port, and dispatches an install_node job. The buyer's site is a container on infrastructure the operator owns.
• customer_cloud: the buyer's site runs on a server in *their own cloud account, billed to them by the provider — see below. A product opts in by picking Customer cloud server in the product-edit Purchase grants picker (CustomerCloudFulfillment, registered with the store's FulfillmentRegistry from serve.php); that stamps the provider value and contributes the domain question as a checkout requirement automatically.

The customer-cloud provisioning keypair (the public half is installed on created instances; the private half is the management node's only access to them) is generated automatically at plugin activation (activate.php → ProvisioningSetup::ensureSshKey()), defaulting to {site root}/config/provisioning_key. The page's Generate provisioning key button runs the same idempotent action for management nodes activated before the key existed; an existing key or custom path is never overwritten.

Customer-Cloud Fulfillment

The buyer connects their Linode account once at /profile/server_manager/connect_cloud (the Connect page — also the re-connect page if a grant is later revoked). The grant flows through the platform OAuth2 core (provider linode, consumer purpose customer_cloud, scope linodes:read_write only — no account or billing access). Tokens are SecretBox-encrypted on the buyer's CustomerCloudAccount row.

Each provision is a CustomerCloudProvision row that the Provision Customer Cloud scheduled task advances:

pending_connect → (grant arrives) → ready → instance created on the connected account → booting → running + IP → ManagedNode + install_node job → installing → done (or failed, which alerts the ops address).

Provisions have two origins (cvp_origin):

• order — created by a customer-cloud purchase. Starts at pending_connect, installs fresh + Docker, and sends the buyer welcome email on completion (the order-item linkage drives it).
• admin — created by the Install New Node form's cloud-instance target. Starts at ready (the admin picked an already-connected account), carries its install parameters on the row (cvp_docker_mode, cvp_install_mode, cvp_source_node_id, cvp_backup_source, cvp_port, cvp_sitename), and sends no welcome email. The row belongs to the grant owner (cvp_usr_user_id), so a stale grant is re-connectable by the person who can actually re-consent.

The buyer legally owns the domain from the moment it is registered. They are the WHOIS registrant on day one; the operator holds only management and billing, so that buying it could be one click. Those move to the buyer later (see Graduation). Ownership is never in question and never waits on a step.

Selling it

Two things have to be set before a domain can be sold, and until both are the checkout field refuses the order rather than taking money for a name it cannot register:

1. Registrar credentials — the Domain registration card on Server Manager → Provisioning. Namecheap needs an API username, an API key (sealed at rest), and the management node's public IPv4 address allowlisted in its API panel. Namecheap grants API access only to accounts with 20+ domains, $50 in the balance, or $50 spent in the last two years. A sandbox switch points every registrar call at Namecheap's sandbox for a full rehearsal.
2. A domain-year product — an ordinary store product, not publicly listed, with one version whose price type is user. Select it in the store's store_domain_registration_product_id setting. Its price comes from the live registrar quote at checkout, so the buyer pays one year at cost.

The mirror case has its own sweep, because there is nothing to run it from. A buyer who removes the hosting line from the cart and keeps the domain line pays for a domain year whose intake never fires — no row is written, so no queue could ever show it. ManagedDomainWatch therefore looks for the arithmetic signature directly: an order with more paid domain-year lines than registration rows. It reports each such order once (a high-water mark over order-item ids), and gives a fresh charge fifteen minutes to file its row before judging it.

A terminal failure parks the row at failed and emails the provisioning alert address. It is never auto-retried: a name someone else took needs a conversation with the buyer, not another attempt.

Ownership and graduation

Legal ownership is immediate and never moves. What moves is custody — rdm_graduation_state, running operator_managed → push_requested → push_sent → self_custody.

While the domain sits in the operator's registrar account, its renewal bills the operator, and the platform never renews a buyer's domain and never fronts the cost. So the domain has to reach the buyer's own account before its first expiry. ManagedDomainWatch is what makes sure it does:

• It sweeps for domain years that were paid for but never registered (above).
• It refreshes the expiry from the registrar at most weekly.
• At expiry minus six months it pushes a custody state to the buyer's box, and ManagedDomainNotice starts rendering a take-ownership notice there — calm at first, sharper at 30, 14, 7 and 1 days. That notice is the buyer's first mention of graduation anywhere; nothing in the setup wizard or the welcome email raises it. It is shown to permission-5+ users only, never to the site's visitors.
• Once a push is in flight it asks inAccount(). False is the success signal — the domain has left the operator's account — and flips the row to self_custody, updates the box, and emails the buyer a confirmation with the auto-renew reminder.

{"access_key": "...", "secret_key": "...", "region": "...", "endpoint": "..."}

Two columns hold two keys: bkt_credentials is the main (delete-capable) credential the management node itself uses, and bkt_node_credentials optionally holds a write-only key handed to nodes instead (see The node may write to the shelf but never erase it). Both are SecretBox-sealed at rest.

A persisted job command never contains a credential — it carries a placeholder token that the agent resolves in memory immediately before the step runs: __SM_CREDS_<target_id>__ for the main slot, __SM_NODE_CREDS_<target_id>__ for the node slot. The builder decides at build time which token a step gets: node-side uploads carry the node token whenever the node slot is filled, while node-side downloads and cloud deletes always carry the main token, because those need the read and delete capability a write-only key deliberately lacks. The agent resolves exactly the slot the token names and never falls back to the other, so a job built against a since-emptied slot fails visibly rather than running with a more powerful key than intended.

For node-side operations (upload, delete, download), the resolved credentials are embedded into a self-contained PHP script that is piped to the node via a heredoc'd php -- invocation — never written to a file on the node and never visible in process listings as positional arguments. The S3Signer.php and node_uploader.php source is composed at job-build time by JobCommandBuilder::build_node_uploader_script().

Because the script is composed from the management node's own copy of those two files, changes to the signer or the uploader reach every node on its next job — there is no agent release or node upgrade in the loop.

Transient Failures

A storage provider that answers a request with a 5xx does not fail the job. S3Signer::request() retries — MAX_ATTEMPTS tries, exponential backoff with jitter — for the failures that are worth another go: 5xx, 429, 408, and the transport errors that mean the connection died rather than the request being wrong. A deterministic error (403 signature, 404, 400) is returned immediately; retrying it would only burn the budget and bury the message. S3Signer::is_retryable() is the whole policy and is pure, so the classification is testable without a network.

Retrying is safe because every request the class makes is idempotent: a PUT overwrites its key (a single PUT, never multipart, so no orphaned parts survive a failure), and GET/DELETE/list have no cumulative effect.

Two bounds keep a retry from doing harm:

• Wall clock. Total time is capped at one attempt's timeout plus RETRY_WINDOW_SECONDS. An attempt that burns the entire transfer timeout leaves no room for another — the right answer, since a transfer that cannot finish in an hour will not finish on the second try. Job steps that shell out to the uploader take their own timeout from S3Signer::transfer_budget_seconds(), so the agent can never kill a transfer part-way through a retry.
• Replay. A retried upload rewinds the body stream before resending. curl consumes the stream on the first attempt while CURLOPT_INFILESIZE still claims the full length, so a retry without the rewind sends nothing and then blocks until the timeout — a hang rather than an error. A stream that cannot seek is not retried at all, rather than being sent truncated.

• recovery — the recovery public key the node itself holds and has verified, read on the node. The private half lives in a password manager, held by whoever administers that node, and never touches a server. An operator who administers several sites may configure the same public key on all of them, and then one private key opens all of theirs; that is their arrangement to make, on each site, and not something this management node can impose from here.
• site — a keypair the node itself holds at config/backup_site_key. This is what lets a site restore itself with nobody present: pre-restore rollback snapshots and routine restores need no operator. It is disposable — lose it and the recovery key still opens everything, and the next run mints a new one.

• The recovery keypair is generated with maintenance_scripts/sysadmin_tools/escrow_keypair.php (standalone PHP + sodium, no platform bootstrap, so it runs during disaster recovery when the platform is gone), or in the browser from the setup panel.
• The public key is stored in the core backup_recovery_public_key setting.
• Minting and sealing happen on the node, in maintenance_scripts/sysadmin_tools/backup_envelope.php. Only the recovery public key travels in the job step, so a ManagementJob row — which persists forever — carries nothing that can open anything.
• The plaintext key exists only as a 0600 file for the length of the run and is shredded by the step that seals the envelope to the finished archive.

FleetBackupPolicy resolves each node's schedule: the declared fleet settings, then that node's own mgn_backup_policy overrides. The fleet default is enabled. That default is what stops a newly managed node being forgotten — there is deliberately no detector for "nobody has decided about this node", because a node nobody decided about is backed up anyway.

The node detail Backups tab edits the policy, as one of three positions:

• Fleet default stores nothing, so the node follows the fleet settings — including future changes to them.
• A schedule of its own stores the full field set (frequency, window, mode, retention, full interval), frozen against the fleet default: a value the operator saw and saved is a value they chose.
• Off stores exactly that decision, which is what lets the dashboard treat a node without fleet backups as somebody's choice rather than a gap.

{
    "steps": [
        {"type": "ssh", "label": "Check disk usage", "cmd": "df -h /"},
        {"type": "ssh", "label": "Check memory", "cmd": "free -m"},
        {"type": "ssh", "label": "Check uptime", "cmd": "uptime"},
        {"type": "ssh", "label": "Check PostgreSQL", "cmd": "pg_isready"},
        {"type": "ssh", "label": "Check Joinery version",
         "cmd": "grep VERSION /var/www/html/site/public_html/includes/version.php"},
        {"type": "ssh", "label": "Container stats",
         "cmd": "docker stats --no-stream empoweredhealthtn", "on_host": true}
    ]
}

A job's steps form two phases. Steps without the teardown flag are the main phase: they run in order, and a hard failure (no continue_on_error) stops the phase and determines the job's outcome. Steps flagged "teardown": true are the teardown phase: they run on every exit path — success, mid-job failure, or none-of-the-main-steps-ran — so the scratch files a job creates (dumps, staged archives, unpacked installers) are removed even when the job aborts on a shared production host.

Teardown semantics:

• Teardown never changes the outcome. A failed job stays failed with the original failing step in mjb_error_message; a teardown step erroring is logged under the === Teardown === output header and ignored.
• Teardown runs before the terminal status is written. The job stays running while teardown executes, so the per-node concurrency lock holds (no re-run can race the deletions) and the job detail view keeps streaming.
• Progress counts main steps only. mjb_total_steps excludes teardown steps and teardown output never advances mjb_current_step.
• Stale-job replay. Jobs force-failed at agent startup (left running by a crash or restart) get their teardown steps replayed from mjb_commands — safe because every teardown command is an idempotent rm on a per-job path.
• Placement. Builders put teardown steps at the tail of the array, after every main step, and keep continue_on_error on them. An agent that ignores the flag runs the array sequentially, so tail placement makes the steps plain trailing cleanup there — correct, just not failure-proof.

// plugins/server_manager/includes/JobCommandBuilder.php
public static function build_restart_apache($node) {
    return [
        ['type' => 'ssh', 'label' => 'Restart Apache',
         'cmd' => 'systemctl restart apache2'],
        ['type' => 'ssh', 'label' => 'Verify Apache status',
         'cmd' => 'systemctl is-active apache2'],
    ];
}

$steps = JobCommandBuilder::build_restart_apache($node);
$job = ManagementJob::createJob($node->key, 'restart_apache', $steps, null, $session->get_user_id());
header('Location: /admin/server_manager/job_detail?job_id=' . $job->key);

$job = ManagementJob::createJob(
    $node_id,               // target node ID (or null for local)
    'backup_database',      // job type label
    $steps,                 // array of step dicts from JobCommandBuilder
    ['encryption' => true], // parameters (stored for reference/re-run)
    $session->get_user_id() // who triggered it
);

Per-node IP pinning. http_status checks pin the request to the node's own mgn_host IP (CURLOPT_RESOLVE, SNI/Host preserved) when mgn_host is an IP literal and appears in the site hostname's public A records (DnsResolver::getA()) — the same directly-exposed guard check_cert_expiry() uses. A node behind a shared or round-robin hostname (e.g. two DNS servers sharing dns.scrolldaddy.app via dual A records) is therefore checked as itself, not whichever A record DNS happens to return — otherwise a single dead node hides behind its live partner. The guard matters: pinning a Cloudflare-fronted node to its origin IP would bypass the edge and hit the Apache default-vhost fallback cert, failing SNI/cert validation and reporting a false down — so nodes that aren't directly exposed are checked unpinned, through the public hostname, same as before pinning existed.

Check types (extensible via a single dispatch switch in RunNodeUptimeChecks::run_check()):

It reads the served certificate over the wire (stream_socket_client on ssl://mgn_host:443, capture_peer_cert, SNI = the site hostname), so it sees whatever the cert manager actually serves — Caddy, certbot, anything. Validity is deliberately not verified, so the notAfter of an already-expired or near-expiry cert is still readable.

It self-limits to self-renewed, directly-exposed nodes via two guards, because probing an origin behind a CDN returns a misleading cert:

1. Directly-exposed: mgn_host must appear in the public A records for the site hostname (DnsResolver::getA()). A Cloudflare-fronted hostname resolves to Cloudflare, not the origin — those are skipped, because Cloudflare renews that edge cert (not our failure surface).
2. SAN match: the served cert's CN/SANs must actually cover the hostname. A shared default-vhost fallback cert (SAN mismatch) is ignored — there is nothing dedicated to monitor.

1. Database restores replace -- A database restore leaves the target equal to the snapshot. Every restore site (restore_database, both copy jobs, the from-backup install) verifies the archive with gunzip -t before anything is destroyed, drops and recreates the public schema so target-only objects are removed too, and loads with psql -v ON_ERROR_STOP=1 so the first load error fails the job instead of completing a partial restore. Dumps are plain pg_dump snapshots -- the restore step owns the replacement guarantee, so it holds for any file it is fed. Job-internal dumps (copy jobs, install clone) add --no-owner --no-acl because they are restored as the target* site's DB user; backup files restore onto the site that made them, where the role matches.

1. Per-node concurrency lock -- The agent skips jobs if another job is already running on the same node, preventing conflicts.

1. Stale job recovery -- On agent startup, any orphaned running jobs are marked failed with a descriptive message.

1. Step timeout -- 30-minute default per step, overridable. On timeout, the SSH session is killed.

1. Single-threaded agent -- One job at a time. Queued jobs run sequentially.

1. Remote credentials at runtime -- Database credentials for backup/copy/restore are extracted from each node's Globalvars_site.php at execution time, never stored on the management node.

API Actions

The dashboard's page JS calls these POST /api/v1/action/server_manager/{name} actions with the browser-session credential (superadmin only, floor 10) and reads the response envelope's data. The full set: probe_api, job_status, discover_nodes, backup_actions, refresh_node_status, add_discovered_nodes.

server_manager/job_status

Polled by the job detail page for live output.

Parameters:

• job_id (int) -- job to query
• output_offset (int) -- character position; only new output since this offset is returned

{
    "success": true,
    "status": "running",
    "new_output": "=== [Step 2/5] Check memory ===\n...",
    "output_offset": 1234,
    "current_step": 2,
    "total_steps": 5,
    "error_message": null
}

The UI polls every 2 seconds while a job is running and stops when status is completed or failed.

server_manager/backup_actions

Used by the backup browser on the Backups tab.

server_manager/discover_nodes

Used by the auto-detect panel on the Add Node page. Creates and polls discover_nodes jobs.

Troubleshooting

Agent shows Offline on dashboard

• Check the agent is running: sudo systemctl status joinery-agent
• Check logs: journalctl -u joinery-agent -f
• Verify DB credentials in /etc/joinery-agent/joinery-agent.env match those in Globalvars_site.php

• Agent is not running or can't connect to the database
• Another job is running on the same node (per-node lock)

• Verify SSH key path on the node record matches an actual key file
• Test manually: ssh -i /path/to/key root@host "echo ok"
• For container nodes, verify the container name is correct

• The agent crashed or was restarted mid-job. Check journalctl for the crash cause.
• The partially-completed job should be inspected manually. Use Re-run to retry.

File Reference

Plugin (plugins/server_manager/)

Go Agent (/home/user1/joinery-agent/)