Server Manager

The Server Manager plugin provides a web UI for managing remote Joinery production servers. Operations include status checks, backups, database copies, and applying updates -- all from the admin interface at /admin/server_manager.

The system has two components:

• PHP plugin (plugins/server_manager/) -- admin UI, job creation, command generation
• Go agent (/home/user1/joinery-agent/) -- generic step executor that polls the job queue and runs commands via SSH

Quick Start

1. Install the plugin

The plugin is already in the plugins/ directory. From the admin panel:

1. Go to /admin/admin_plugins
2. Click Actions on "Server Manager" and choose Install
3. Click Actions again and choose Activate

2. Install and start the Go agent

Build

cd /home/user1/joinery-agent
make release VERSION=1.0.0

This compiles the binary and packages it into joinery-agent-installer.sh — a self-extracting script that handles both fresh installs and upgrades.

Install

sudo bash joinery-agent-installer.sh --verbose [--config /path/to/Globalvars_site.php]

The installer detects the host's supervision capability:

• systemd hosts: installs /etc/systemd/system/joinery-agent.service; start with systemctl start joinery-agent.
• No systemd (Docker containers, minimal hosts): installs /usr/local/bin/joinery-agent-supervise plus /etc/cron.d/joinery-agent (@reboot + a once-a-minute keepalive) and starts the agent immediately. Logs go to /var/log/joinery-agent.log.

• /usr/local/bin/joinery-agent — the binary
• /etc/joinery-agent/joinery-agent.env — configuration (from example, first install only)

Every control plane needs a live agent. All jobs (install_node, provision_ssl, backups, upgrades) sit pending until an agent polling that site's own database claims them. The Server Manager → Provisioning page shows an agent heartbeat badge as requirement #1.

Configure (usually not needed)

The agent reads database credentials directly from Globalvars_site.php — no manual configuration required on a standard Joinery install.

The default config path is /var/www/html/joinerytest/config/Globalvars_site.php. If your install is at a different path, set it in the env file:

sudo nano /etc/joinery-agent/joinery-agent.env
# Set: JOINERY_CONFIG=/var/www/html/mysite/config/Globalvars_site.php

Other optional settings in the env file:

Start

sudo systemctl start joinery-agent
sudo systemctl status joinery-agent

The dashboard at /admin/server_manager should now show Agent Status: Online.

If anything is wrong, the agent logs to the systemd journal (systemd hosts) or /var/log/joinery-agent.log (cron-supervised hosts):

journalctl -u joinery-agent -f      # systemd
tail -f /var/log/joinery-agent.log  # cron supervision

Common startup errors are self-explanatory — missing DB_NAME, wrong password, or plugin tables not installed. Each error message tells you exactly what to fix.

Upgrade

cd /home/user1/joinery-agent
make release VERSION=1.x.x
sudo bash joinery-agent-installer.sh --verbose

The installer auto-detects upgrades: stops the service, swaps the binary, restarts, and rolls back automatically if the new version fails to start.

3. Add managed nodes

Go to /admin/server_manager/node_add (or click Add Node on the dashboard). There are two ways to add nodes:

Auto-detect (recommended)

The Auto-Detect Joinery Servers panel scans a remote host for Joinery instances automatically. Enter:

1. SSH Host -- the server IP (e.g., 23.239.11.53)
2. SSH Key Path -- path to the private key on the control plane (defaults to /home/user1/.ssh/id_ed25519_claude)
3. Click Detect

Detected instances appear as cards with Add This Node buttons. Clicking one auto-fills the entire form below -- just click Add Node to save.

Auto-detect requires the Go agent to be running (it executes the SSH commands, not PHP).

Manual

Fill in the form fields directly:

Admin Pages

All pages are at /admin/server_manager/... and require permission level 10 (superadmin).

The UI is organized around a dashboard + node detail pattern. The dashboard shows the fleet overview; clicking a node opens a tabbed detail page with all operations for that node.

Node Detail Tabs

The node detail page (/admin/server_manager/node_detail?mgn_id=N&tab=...) has five tabs:

Dashboard Features

The dashboard shows:

• Agent Status -- online/offline indicator with version and last heartbeat time
• Managed Nodes -- cards with health-based status dots (green=healthy, yellow=warning, red=problem, gray=no data), key metrics, and action buttons
• Publish Upgrade -- build upgrade archives from control plane source code (node-independent)
• Recent Jobs -- latest 20 jobs across all nodes

• Red: Last check failed, disk > 90%, or PostgreSQL not accepting connections
• Yellow: Disk > 80% or load average > 5
• Green: All metrics healthy
• Gray: Never checked or no data

Job Types

Note on bare-metal nodes with user1 SSH: When a bare-metal install completes, install.sh disables root SSH and the node's mgn_ssh_user is automatically updated to user1. Subsequent jobs run as user1 with NOPASSWD sudo. All backup/restore commands that need root-level paths (e.g. /backups/) use sudo automatically.

One-Click Node Install

Dashboard → Install New Node opens a form that provisions a Joinery site in a single click. The Target Host dropdown offers three kinds of target:

• A known host (or Other server with manual SSH details): the form creates the ManagedNode and dispatches the install_node job immediately.
• Create a new cloud instance: no server exists yet. The form records an admin-origin CustomerCloudProvision (connected cloud account, region, instance type, plus all install parameters) and the Provision Customer Cloud task births the instance, creates the node, and dispatches the install — see Customer-Cloud Fulfillment. The instance is created in, and billed to, the selected connected account; Linode grants expire after two hours, so connect (or re-connect) shortly before submitting. Cloud targets always take a fresh source backup in From-Backup mode. In-flight provisions appear in a banner at the top of the dashboard.

Two install types:

• Fresh: empty Joinery site with default schema. Admin picks the domain. Default admin login is [email protected] / changeme123, with usr_force_password_change=true so the first login forces a new password.
• From Backup: fresh install + restore of a source node's DB and project files. Target inherits the source's domain — admin cuts over DNS after install. Use source admin credentials to log in.

From-Backup restores files by extracting the source archive with both of its leading path components stripped, taking only the project_files/ subtree — backup_project.sh writes archives as `{backup_name}/project_files/{public_html, uploads,config,...} with the archive's own metadata (apache_config/`, backup_info.txt, the .sql dump) as siblings. The target keeps its own Globalvars_site.php. A verification step then requires every regular file the archive carries to exist at the site root and fails the job otherwise, because a clone whose files did not land still serves pages: the fresh install ran first and the database restore succeeded, so the only symptom is uploaded files missing from where the restored database says they are.

The mgn_install_state column tracks the lifecycle: installing → NULL (success) or install_failed (failure). On failure, the node detail page surfaces a Retry Install button; the target must be cleaned manually (e.g. rm -rf /var/www/html/SITENAME) before retry because install.sh refuses to overwrite an existing site. Postgres passwords are auto-generated and stored in the target's Globalvars_site.php — Server Manager does not capture or display them.

Docker notes:

• The reverse proxy step (manage_domain.sh) is skipped when the domain is a bare IP address — a routable hostname is required for Apache ServerName-based virtual hosting. With an IP domain, the site is accessible directly on its mapped port.
• backup_project.sh requires rsync. The bare-metal and Docker install scripts install rsync as part of the essential packages (install.sh line ~948). Sites installed before this was added can install it manually with apt install rsync.
• After a Docker install, mgn_container_name is automatically recorded in the control plane DB so future jobs correctly use docker exec to reach the site.

SSL Management

SSL State

Each node tracks its TLS certificate state in mgn_ssl_state:

Automatic Detection

check_status jobs include an SSH step that checks for a Let's Encrypt cert under /etc/letsencrypt/live/{domain}/. JobResultProcessor updates mgn_ssl_state and stores ssl_domain, ssl_expiry_raw, and ssl_expiry_ts in mgn_last_status_data. State transitions:

• CERT_FOUND → sets state to active (from any prior state)
• CERT_MISSING → clears state to null only if currently null or active; never overwrites pending or failed

Manual Provisioning

The Overview tab shows an SSL Setup card when mgn_ssl_state is not active, the node has a domain in its site URL, and mgn_cert_expiry_ts is empty. The last condition excludes directly-exposed, self-renewed nodes (see Certificate Expiry Monitoring) — their cert lifecycle is owned by an external renewer (e.g. Caddy), and the card's certbot-based provisioning does not apply to them. The card:

1. Resolves the domain via DNS and shows whether it points to the node's host IP
2. Enables the Provision SSL button when DNS is ready (or when the host IP is not configured)
3. On submit: creates a provision_ssl job, sets mgn_ssl_state = 'pending', redirects to job detail

Automated Provisioning (installs only)

For nodes installed via Install New Node, ProvisionPendingSsl (scheduled hourly) watches for nodes with mgn_ssl_state = 'pending', checks DNS, and kicks off provision_ssl jobs automatically. After ~16 hours of DNS check failures it flips state to failed. Manual provisioning via the Setup card is the fallback.

Hosting Provisioning

Paid hosting orders on getjoinery become installed, SSL'd Joinery sites with no human touch. The Poll Hosting Orders scheduled task polls the getjoinery API each cron tick for paid orders carrying an answer to the configured domain Question, and fulfills each one in the mode the product declares in pro_fulfillment_provider:

• Shared host (the default, any other value): the pipeline picks the least-loaded provisioning-enabled ManagedHost, assigns the next Docker port, and dispatches an install_node job. The buyer's site is a container on infrastructure the operator owns.
• customer_cloud: the buyer's site runs on a server in *their own cloud account, billed to them by the provider — see below. A product opts in by picking Customer cloud server in the product-edit Purchase grants picker (CustomerCloudFulfillment, registered with the store's FulfillmentRegistry from serve.php); that stamps the provider value and contributes the domain question as a checkout requirement automatically.

The customer-cloud provisioning keypair (the public half is installed on created instances; the private half is the control plane's only access to them) is generated automatically at plugin activation (activate.php → ProvisioningSetup::ensureSshKey()), defaulting to {site root}/config/provisioning_key. The page's Generate provisioning key button runs the same idempotent action for control planes activated before the key existed; an existing key or custom path is never overwritten.

Customer-Cloud Fulfillment

The buyer connects their Linode account once at /profile/server_manager/connect_cloud (the Connect page — also the re-connect page if a grant is later revoked). The grant flows through the platform OAuth2 core (provider linode, consumer purpose customer_cloud, scope linodes:read_write only — no account or billing access). Tokens are SecretBox-encrypted on the buyer's CustomerCloudAccount row.

Each provision is a CustomerCloudProvision row that the Provision Customer Cloud scheduled task advances:

pending_connect → (grant arrives) → ready → instance created on the connected account → booting → running + IP → ManagedNode + install_node job → installing → done (or failed, which alerts the ops address).

Provisions have two origins (cvp_origin):

• order — created by a customer-cloud purchase. Starts at pending_connect, installs fresh + Docker, and sends the buyer welcome email on completion (the order-item linkage drives it).
• admin — created by the Install New Node form's cloud-instance target. Starts at ready (the admin picked an already-connected account), carries its install parameters on the row (cvp_docker_mode, cvp_install_mode, cvp_source_node_id, cvp_backup_source, cvp_port, cvp_sitename), and sends no welcome email. The row belongs to the grant owner (cvp_usr_user_id), so a stale grant is re-connectable by the person who can actually re-consent.

{"access_key": "...", "secret_key": "...", "region": "...", "endpoint": "..."}

cat ~/.joinery_backup_key

{
    "steps": [
        {"type": "ssh", "label": "Check disk usage", "cmd": "df -h /"},
        {"type": "ssh", "label": "Check memory", "cmd": "free -m"},
        {"type": "ssh", "label": "Check uptime", "cmd": "uptime"},
        {"type": "ssh", "label": "Check PostgreSQL", "cmd": "pg_isready"},
        {"type": "ssh", "label": "Check Joinery version",
         "cmd": "grep VERSION /var/www/html/site/public_html/includes/version.php"},
        {"type": "ssh", "label": "Container stats",
         "cmd": "docker stats --no-stream empoweredhealthtn", "on_host": true}
    ]
}

A job's steps form two phases. Steps without the teardown flag are the main phase: they run in order, and a hard failure (no continue_on_error) stops the phase and determines the job's outcome. Steps flagged "teardown": true are the teardown phase: they run on every exit path — success, mid-job failure, or none-of-the-main-steps-ran — so the scratch files a job creates (dumps, staged archives, unpacked installers) are removed even when the job aborts on a shared production host.

Teardown semantics:

• Teardown never changes the outcome. A failed job stays failed with the original failing step in mjb_error_message; a teardown step erroring is logged under the === Teardown === output header and ignored.
• Teardown runs before the terminal status is written. The job stays running while teardown executes, so the per-node concurrency lock holds (no re-run can race the deletions) and the job detail view keeps streaming.
• Progress counts main steps only. mjb_total_steps excludes teardown steps and teardown output never advances mjb_current_step.
• Stale-job replay. Jobs force-failed at agent startup (left running by a crash or restart) get their teardown steps replayed from mjb_commands — safe because every teardown command is an idempotent rm on a per-job path.
• Placement. Builders put teardown steps at the tail of the array, after every main step, and keep continue_on_error on them. An agent that ignores the flag runs the array sequentially, so tail placement makes the steps plain trailing cleanup there — correct, just not failure-proof.

// plugins/server_manager/includes/JobCommandBuilder.php
public static function build_restart_apache($node) {
    return [
        ['type' => 'ssh', 'label' => 'Restart Apache',
         'cmd' => 'systemctl restart apache2'],
        ['type' => 'ssh', 'label' => 'Verify Apache status',
         'cmd' => 'systemctl is-active apache2'],
    ];
}

$steps = JobCommandBuilder::build_restart_apache($node);
$job = ManagementJob::createJob($node->key, 'restart_apache', $steps, null, $session->get_user_id());
header('Location: /admin/server_manager/job_detail?job_id=' . $job->key);

$job = ManagementJob::createJob(
    $node_id,               // target node ID (or null for local)
    'backup_database',      // job type label
    $steps,                 // array of step dicts from JobCommandBuilder
    ['encryption' => true], // parameters (stored for reference/re-run)
    $session->get_user_id() // who triggered it
);

Per-node IP pinning. http_status checks pin the request to the node's own mgn_host IP (CURLOPT_RESOLVE, SNI/Host preserved) when mgn_host is an IP literal and appears in the site hostname's public A records (DnsResolver::getA()) — the same directly-exposed guard check_cert_expiry() uses. A node behind a shared or round-robin hostname (e.g. two DNS servers sharing dns.scrolldaddy.app via dual A records) is therefore checked as itself, not whichever A record DNS happens to return — otherwise a single dead node hides behind its live partner. The guard matters: pinning a Cloudflare-fronted node to its origin IP would bypass the edge and hit the Apache default-vhost fallback cert, failing SNI/cert validation and reporting a false down — so nodes that aren't directly exposed are checked unpinned, through the public hostname, same as before pinning existed.

Check types (extensible via a single dispatch switch in RunNodeUptimeChecks::run_check()):

It reads the served certificate over the wire (stream_socket_client on ssl://mgn_host:443, capture_peer_cert, SNI = the site hostname), so it sees whatever the cert manager actually serves — Caddy, certbot, anything. Validity is deliberately not verified, so the notAfter of an already-expired or near-expiry cert is still readable.

It self-limits to self-renewed, directly-exposed nodes via two guards, because probing an origin behind a CDN returns a misleading cert:

1. Directly-exposed: mgn_host must appear in the public A records for the site hostname (DnsResolver::getA()). A Cloudflare-fronted hostname resolves to Cloudflare, not the origin — those are skipped, because Cloudflare renews that edge cert (not our failure surface).
2. SAN match: the served cert's CN/SANs must actually cover the hostname. A shared default-vhost fallback cert (SAN mismatch) is ignored — there is nothing dedicated to monitor.

1. Database restores replace -- A database restore leaves the target equal to the snapshot. Every restore site (restore_database, both copy jobs, the from-backup install) verifies the archive with gunzip -t before anything is destroyed, drops and recreates the public schema so target-only objects are removed too, and loads with psql -v ON_ERROR_STOP=1 so the first load error fails the job instead of completing a partial restore. Dumps are plain pg_dump snapshots -- the restore step owns the replacement guarantee, so it holds for any file it is fed. Job-internal dumps (copy jobs, install clone) add --no-owner --no-acl because they are restored as the target* site's DB user; backup files restore onto the site that made them, where the role matches.

1. Per-node concurrency lock -- The agent skips jobs if another job is already running on the same node, preventing conflicts.

1. Stale job recovery -- On agent startup, any orphaned running jobs are marked failed with a descriptive message.

1. Step timeout -- 30-minute default per step, overridable. On timeout, the SSH session is killed.

1. Single-threaded agent -- One job at a time. Queued jobs run sequentially.

1. Remote credentials at runtime -- Database credentials for backup/copy/restore are extracted from each node's Globalvars_site.php at execution time, never stored on the control plane.

API Actions

The dashboard's page JS calls these POST /api/v1/action/server_manager/{name} actions with the browser-session credential (superadmin only, floor 10) and reads the response envelope's data. The full set: probe_api, job_status, discover_nodes, backup_actions, refresh_node_status, add_discovered_nodes.

server_manager/job_status

Polled by the job detail page for live output.

Parameters:

• job_id (int) -- job to query
• output_offset (int) -- character position; only new output since this offset is returned

{
    "success": true,
    "status": "running",
    "new_output": "=== [Step 2/5] Check memory ===\n...",
    "output_offset": 1234,
    "current_step": 2,
    "total_steps": 5,
    "error_message": null
}

The UI polls every 2 seconds while a job is running and stops when status is completed or failed.

server_manager/backup_actions

Used by the backup browser on the Backups tab.

server_manager/discover_nodes

Used by the auto-detect panel on the Add Node page. Creates and polls discover_nodes jobs.

Troubleshooting

Agent shows Offline on dashboard

• Check the agent is running: sudo systemctl status joinery-agent
• Check logs: journalctl -u joinery-agent -f
• Verify DB credentials in /etc/joinery-agent/joinery-agent.env match those in Globalvars_site.php

• Agent is not running or can't connect to the database
• Another job is running on the same node (per-node lock)

• Verify SSH key path on the node record matches an actual key file
• Test manually: ssh -i /path/to/key root@host "echo ok"
• For container nodes, verify the container name is correct

• The agent crashed or was restarted mid-job. Check journalctl for the crash cause.
• The partially-completed job should be inspected manually. Use Re-run to retry.

File Reference

Plugin (plugins/server_manager/)

Go Agent (/home/user1/joinery-agent/)