Hub stage 2: install, update and uninstall apps on several servers

Writes use only <ID>_ADMIN_TOKEN and need the apps.write capability.
Store URLs are resolved from each server's own listing. The config is saved
to history before an uninstall. Zip uploads are fanned out to the servers.
Every target's result is audited. The console gets a row menu on the
matrix, a target picker, a results dialog with Retry failed, the store and
install from file in All servers mode, and single-server actions for writers.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

hub/README.md

@@ -8,7 +8,7 @@ read-only, Technitium-compatible `/api/*`:
 - No `node` (or `node=cluster`) asks every server in parallel and merges the answers
   (statistics are summed, top lists and query logs are merged). If a server does not
   answer, the others are shown and the response carries `hubWarnings`.
-- Anything that would change data is refused with `403` (and written to the audit log). The hub has its own accounts with roles, a login rate limit and an audit log.
+- Reads go with each server's read token. The only writes are the app changes below, which use a separate admin token and a role with `apps.write`. Anything else that would change data is refused with `403` (and written to the audit log). The hub has its own accounts with roles, a login rate limit and an audit log.
 
 The servers themselves are not changed: each only needs a read-only API token. Create it
 for a user whose group has View, and nothing else, on Dashboard, Logs, Apps (the logging
@@ -23,7 +23,7 @@ apps Query Logs reads from) and Settings (the blocking state).
 | `HUB_USER`, `HUB_PASSWORD` | see Users | Bootstrap: one `admin`, used only while `${HUB_DATA}/users.json` does not exist. Set both or neither. |
 | `HUB_DATA` | `/data` | Folder of `users.json` and `audit.log`. Mount a volume on it. |
 | `HUB_TRUST_PROXY` | off | `1` believes the first `X-Forwarded-For` entry as the client address (set it behind Traefik). Leave it off when clients reach the hub directly, or anyone can fake their address. |
-| `<ID>_ADMIN_TOKEN` | optional | A write-capable token per server (`MDNS_ADMIN_TOKEN`). Read into the config for the coming write stages; not used yet and never logged. |
+| `<ID>_ADMIN_TOKEN` | optional | A write-capable token per server (`MDNS_ADMIN_TOKEN`). Only app writes use it, and only it: a server without one is read-only for the hub (the matrix and the session list the writable servers). Never logged or returned. |
 | `HUB_PORT` | `8080` | Listening port. |
 | `HUB_CACHE_SECONDS` | `15` | How long successful answers of the servers are cached. |
 | `HUB_TIMEOUT_MS` | `8000` | Per-request timeout towards a server. |
@@ -57,7 +57,7 @@ atomically with mode 0600. The running hub notices a change on the next request
 | `apps-admin` | read, apps.write |
 | `admin` | everything, including audit.read |
 
-Only reads exist today; the write capabilities are the machinery for the next stages. Every response also carries
+`apps.write` is what the app writes below need; `operate` and the rest are still unused. Every response also carries
 `hubRole` (and `info.hubRole`).
 
 ## Login protection
@@ -83,7 +83,27 @@ Public: `status`, `user/login`. After signing in: `user/session/get`, `user/logo
 `logs/query`, `logs/list`, `settings/get`, `admin/cluster/state` (synthesized), and the hub's own `hub/audit/list` (role with
 `audit.read`). They are listed in `routes.ts` with a kind (`read`, `write-one`, `write-all`) and the capability they need.
 
-Everything else answers `403 The hub is read-only.` and is audited.
+Everything else answers `403 The hub is read-only.` and is audited. `apps/config/get` and `apps/config/set` are not available yet.
+
+## App writes
+
+Needs the `apps.write` role (`apps-admin`, `admin`); others get `403 Your role does not allow this.` and a `denied-write` audit line.
+Every write goes to a server with its `<ID>_ADMIN_TOKEN`; a server without one answers "no admin token configured" and is not called.
+
+- `POST hub/apps/run` (form or JSON): `action` = `install-store` | `update-store` | `uninstall`, `name`, `targets` (list or
+  comma-separated server ids). Runs on every target in parallel and answers `{ results: [{ server, ok, message? }] }`.
+  The download URL is taken from each target's own `apps/listStoreApps`; a `url` field is only accepted when it matches.
+- `POST hub/apps/upload` (multipart, at most 20 MB, kept in memory only): `action` = `install` | `update`, `name`, `targets`
+  (comma-separated), file field `fileAppZip`. The same zip is forwarded to every target.
+- `apps/downloadAndInstall`, `apps/downloadAndUpdate`, `apps/uninstall`, `apps/install`, `apps/update` with `node=<id>`:
+  the console's own single-server calls, forwarded to that server (store URLs checked as above). Refused without a node, and
+  with `node=cluster`.
+- `apps/listStoreApps`: the first server's store, each app tagged with `installedOn`.
+
+Before an uninstall the app's config is saved to `${HUB_DATA}/apps/<app>/history/<time>__<server>__before-uninstall.txt`
+(50 files per app are kept). If that cannot be done the uninstall does not happen. Every target's result is written to the
+audit log as `apps.<action>`; the history files hold configs, so protect the data folder like the users file.
+After any write the hub forgets its cached answers of that server.
 
 ## Run locally