Hub stage 2: install, update and uninstall apps on several servers

Writes use only <ID>_ADMIN_TOKEN and need the apps.write capability.
Store URLs are resolved from each server's own listing. The config is saved
to history before an uninstall. Zip uploads are fanned out to the servers.
Every target's result is audited. The console gets a row menu on the
matrix, a target picker, a results dialog with Retry failed, the store and
install from file in All servers mode, and single-server actions for writers.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

hub/README.md

@@ -8,7 +8,7 @@ read-only, Technitium-compatible `/api/*`:
 - No `node` (or `node=cluster`) asks every server in parallel and merges the answers
   (statistics are summed, top lists and query logs are merged). If a server does not
   answer, the others are shown and the response carries `hubWarnings`.
-- Anything that would change data is refused with `403` (and written to the audit log). The hub has its own accounts with roles, a login rate limit and an audit log.
+- Reads go with each server's read token. The only writes are the app changes below, which use a separate admin token and a role with `apps.write`. Anything else that would change data is refused with `403` (and written to the audit log). The hub has its own accounts with roles, a login rate limit and an audit log.
 
 The servers themselves are not changed: each only needs a read-only API token. Create it
 for a user whose group has View, and nothing else, on Dashboard, Logs, Apps (the logging
@@ -23,7 +23,7 @@ apps Query Logs reads from) and Settings (the blocking state).
 | `HUB_USER`, `HUB_PASSWORD` | see Users | Bootstrap: one `admin`, used only while `${HUB_DATA}/users.json` does not exist. Set both or neither. |
 | `HUB_DATA` | `/data` | Folder of `users.json` and `audit.log`. Mount a volume on it. |
 | `HUB_TRUST_PROXY` | off | `1` believes the first `X-Forwarded-For` entry as the client address (set it behind Traefik). Leave it off when clients reach the hub directly, or anyone can fake their address. |
-| `<ID>_ADMIN_TOKEN` | optional | A write-capable token per server (`MDNS_ADMIN_TOKEN`). Read into the config for the coming write stages; not used yet and never logged. |
+| `<ID>_ADMIN_TOKEN` | optional | A write-capable token per server (`MDNS_ADMIN_TOKEN`). Only app writes use it, and only it: a server without one is read-only for the hub (the matrix and the session list the writable servers). Never logged or returned. |
 | `HUB_PORT` | `8080` | Listening port. |
 | `HUB_CACHE_SECONDS` | `15` | How long successful answers of the servers are cached. |
 | `HUB_TIMEOUT_MS` | `8000` | Per-request timeout towards a server. |
@@ -57,7 +57,7 @@ atomically with mode 0600. The running hub notices a change on the next request
 | `apps-admin` | read, apps.write |
 | `admin` | everything, including audit.read |
 
-Only reads exist today; the write capabilities are the machinery for the next stages. Every response also carries
+`apps.write` is what the app writes below need; `operate` and the rest are still unused. Every response also carries
 `hubRole` (and `info.hubRole`).
 
 ## Login protection
@@ -83,7 +83,27 @@ Public: `status`, `user/login`. After signing in: `user/session/get`, `user/logo
 `logs/query`, `logs/list`, `settings/get`, `admin/cluster/state` (synthesized), and the hub's own `hub/audit/list` (role with
 `audit.read`). They are listed in `routes.ts` with a kind (`read`, `write-one`, `write-all`) and the capability they need.
 
-Everything else answers `403 The hub is read-only.` and is audited.
+Everything else answers `403 The hub is read-only.` and is audited. `apps/config/get` and `apps/config/set` are not available yet.
+
+## App writes
+
+Needs the `apps.write` role (`apps-admin`, `admin`); others get `403 Your role does not allow this.` and a `denied-write` audit line.
+Every write goes to a server with its `<ID>_ADMIN_TOKEN`; a server without one answers "no admin token configured" and is not called.
+
+- `POST hub/apps/run` (form or JSON): `action` = `install-store` | `update-store` | `uninstall`, `name`, `targets` (list or
+  comma-separated server ids). Runs on every target in parallel and answers `{ results: [{ server, ok, message? }] }`.
+  The download URL is taken from each target's own `apps/listStoreApps`; a `url` field is only accepted when it matches.
+- `POST hub/apps/upload` (multipart, at most 20 MB, kept in memory only): `action` = `install` | `update`, `name`, `targets`
+  (comma-separated), file field `fileAppZip`. The same zip is forwarded to every target.
+- `apps/downloadAndInstall`, `apps/downloadAndUpdate`, `apps/uninstall`, `apps/install`, `apps/update` with `node=<id>`:
+  the console's own single-server calls, forwarded to that server (store URLs checked as above). Refused without a node, and
+  with `node=cluster`.
+- `apps/listStoreApps`: the first server's store, each app tagged with `installedOn`.
+
+Before an uninstall the app's config is saved to `${HUB_DATA}/apps/<app>/history/<time>__<server>__before-uninstall.txt`
+(50 files per app are kept). If that cannot be done the uninstall does not happen. Every target's result is written to the
+audit log as `apps.<action>`; the history files hold configs, so protect the data folder like the users file.
+After any write the hub forgets its cached answers of that server.
 
 ## Run locally
 

hub/apps-matrix.test.ts

@@ -183,7 +183,7 @@ describe('hub/apps/matrix', () => {
     const { body } = await matrix()
     expect(body.status).toBe('ok')
     expect(body.hubWarnings).toEqual([])
-    expect(body.response.servers).toEqual([{ id: 'a', ok: true }, { id: 'b', ok: true }])
+    expect(body.response.servers).toEqual([{ id: 'a', ok: true, writable: false }, { id: 'b', ok: true, writable: false }])
     expect((body.response.apps as any[]).map((x) => x.name)).toEqual(['Differs', 'Empty', 'Only A', 'Plain', 'Same'])
 
     const same = byName(body, 'Same')
@@ -246,7 +246,7 @@ describe('hub/apps/matrix', () => {
     b.down = true
     const { body } = await matrix()
     expect(body.status).toBe('ok')
-    expect(body.response.servers[0]).toEqual({ id: 'a', ok: true })
+    expect(body.response.servers[0]).toEqual({ id: 'a', ok: true, writable: false })
     expect(body.response.servers[1]).toMatchObject({ id: 'b', ok: false })
     expect(body.response.servers[1].error).toBeTruthy()
     expect(body.hubWarnings).toHaveLength(1)
@@ -272,9 +272,9 @@ describe('hub/apps/matrix', () => {
 })
 
 describe('session permissions', () => {
-  it('lets every role view Apps without modifying it', async () => {
+  it('lets the admin view and modify Apps (the other roles are in apps-write.test.ts)', async () => {
     const res = await fetch(`${base}/api/user/session/get`, { headers: { authorization: `Bearer ${token}` } })
     const info = ((await res.json()) as any).info
-    expect(info.permissions.Apps).toEqual({ canView: true, canModify: false, canDelete: false })
+    expect(info.permissions.Apps).toEqual({ canView: true, canModify: true, canDelete: true })
   })
 })

hub/apps-matrix.ts

@@ -29,7 +29,7 @@ export interface MatrixApp {
 }
 
 export interface Matrix {
-  servers: { id: string; ok: boolean; error?: string }[]
+  servers: { id: string; ok: boolean; writable: boolean; error?: string }[]
   apps: MatrixApp[]
 }
 
@@ -128,7 +128,9 @@ export async function buildMatrix(config: HubConfig, upstream: Upstream): Promis
     }),
   )
 
-  const servers = perServer.map((p) => (p.error === undefined ? { id: p.server.id, ok: true } : { id: p.server.id, ok: false, error: p.error }))
+  const servers = perServer.map((p) => (p.error === undefined
+      ? { id: p.server.id, ok: true, writable: p.server.adminToken !== undefined }
+      : { id: p.server.id, ok: false, writable: p.server.adminToken !== undefined, error: p.error }))
   const warnings = perServer.filter((p) => p.error !== undefined).map((p) => `${p.server.id}: ${p.error}`)
   if (warnings.length === perServer.length) throw new Error(`No server answered. ${warnings.join('; ')}`.trim())
 

hub/apps-write.test.ts

@@ -0,0 +1,513 @@
+// @vitest-environment node
+import http from 'node:http'
+import type { AddressInfo } from 'node:net'
+import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import path from 'node:path'
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'
+import { parseConfig } from './config.ts'
+import { createHub } from './server.ts'
+import { boundaryOf, parseMultipart } from './multipart.ts'
+import { safeName } from './apps-write.ts'
+import { addUser, writeUsersFile } from './users.ts'
+
+interface Call {
+  path: string
+  token: string
+  method: string
+  query: URLSearchParams
+  body: Buffer
+  contentType: string
+}
+
+interface Fake {
+  server: http.Server
+  url: string
+  apps: Record<string, { version: string; config: string | null }>
+  store: { name: string; version: string; url: string; description: string; size: string }[]
+  calls: Call[]
+  failWrites: boolean
+  denyConfig: boolean
+}
+
+function fake(): Promise<Fake> {
+  return new Promise((resolve) => {
+    const f: Fake = { server: http.createServer(), url: '', apps: {}, store: [], calls: [], failWrites: false, denyConfig: false }
+    f.server.on('request', (req, res) => {
+      const chunks: Buffer[] = []
+      req.on('data', (c: Buffer) => chunks.push(c))
+      req.on('end', () => {
+        const url = new URL(req.url ?? '/', 'http://fake')
+        const apiPath = url.pathname.replace(/^\/api\//, '')
+        const token = url.searchParams.get('token') ?? ''
+        const name = url.searchParams.get('name') ?? ''
+        f.calls.push({ path: apiPath, token, method: req.method ?? '', query: url.searchParams, body: Buffer.concat(chunks), contentType: String(req.headers['content-type'] ?? '') })
+        res.setHeader('content-type', 'application/json')
+        const ok = (response: unknown = {}) => res.end(JSON.stringify({ status: 'ok', response }))
+        const error = (m: string) => res.end(JSON.stringify({ status: 'error', errorMessage: m }))
+        const isWrite = ['apps/downloadAndInstall', 'apps/downloadAndUpdate', 'apps/uninstall', 'apps/install', 'apps/update'].includes(apiPath)
+        if (isWrite && !token.startsWith('admin-')) return error("Access was denied to user 'reader' for apps.")
+        if (isWrite && f.failWrites) return error('Disk full.')
+        switch (apiPath) {
+          case 'apps/list':
+            return ok({ apps: Object.entries(f.apps).map(([n, a]) => ({ name: n, description: `${n} d`, version: a.version })) })
+          case 'apps/listStoreApps':
+            return ok({ storeApps: f.store })
+          case 'apps/config/get':
+            if (f.denyConfig) return error('Access was denied.')
+            return f.apps[name] ? ok({ config: f.apps[name].config }) : error('not found')
+          case 'apps/downloadAndInstall':
+          case 'apps/downloadAndUpdate': {
+            const entry = f.store.find((s) => s.name === name && s.url === url.searchParams.get('url'))
+            if (!entry) return error('not in store')
+            f.apps[name] = { version: entry.version, config: f.apps[name]?.config ?? null }
+            return ok()
+          }
+          case 'apps/uninstall':
+            delete f.apps[name]
+            return ok()
+          case 'apps/install':
+          case 'apps/update':
+            f.apps[name] = { version: 'zip', config: null }
+            return ok()
+          default:
+            return ok()
+        }
+      })
+    })
+    f.server.listen(0, '127.0.0.1', () => {
+      f.url = `http://127.0.0.1:${(f.server.address() as AddressInfo).port}`
+      resolve(f)
+    })
+  })
+}
+
+let a: Fake
+let b: Fake
+let dataDir: string
+let staticDir: string
+let hub: http.Server | undefined
+let base: string
+const tokens: Record<string, string> = {}
+
+const STORE = (v: string) => [{ name: 'Blocker', version: v, url: `https://store.example/Blocker-v${v}.zip`, description: 'd', size: '1 KB' }]
+
+function reset(): void {
+  for (const f of [a, b]) {
+    f.calls = []
+    f.failWrites = false
+    f.denyConfig = false
+    f.store = STORE('2.0')
+    f.apps = { Blocker: { version: '1.0', config: '{"secret":"cfg"}' } }
+  }
+}
+
+async function start(extra: Record<string, string> = {}): Promise<void> {
+  hub?.close()
+  hub = createHub(
+    parseConfig({
+      HUB_SERVERS: `a=${a.url},b=${b.url}`,
+      A_TOKEN: 'read-a',
+      B_TOKEN: 'read-b',
+      A_ADMIN_TOKEN: 'admin-a',
+      B_ADMIN_TOKEN: 'admin-b',
+      HUB_DATA: dataDir,
+      HUB_STATIC: staticDir,
+      HUB_CACHE_SECONDS: '0',
+      HUB_TIMEOUT_MS: '2000',
+      ...extra,
+    }),
+  )
+  await new Promise<void>((resolve) => hub!.listen(0, '127.0.0.1', resolve))
+  base = `http://127.0.0.1:${(hub.address() as AddressInfo).port}`
+  for (const role of ['viewer', 'operator', 'apps-admin', 'admin']) {
+    const res = await fetch(`${base}/api/user/login`, {
+      method: 'POST',
+      headers: { 'content-type': 'application/x-www-form-urlencoded' },
+      body: `user=${role}&pass=password-${role}`,
+    })
+    tokens[role] = ((await res.json()) as { token: string }).token
+  }
+}
+
+async function api(role: string, apiPath: string, init: { params?: Record<string, string>; method?: string; json?: unknown; body?: string | FormData; headers?: Record<string, string> } = {}) {
+  const query = new URLSearchParams(init.params ?? {})
+  const headers: Record<string, string> = { authorization: `Bearer ${tokens[role]}`, ...(init.headers ?? {}) }
+  let body = init.body
+  if (init.json !== undefined) {
+    headers['content-type'] = 'application/json'
+    body = JSON.stringify(init.json)
+  }
+  const res = await fetch(`${base}/api/${apiPath}?${query}`, { method: init.method ?? (body === undefined ? 'GET' : 'POST'), headers, body })
+  return { code: res.status, body: (await res.json()) as Record<string, any> }
+}
+
+const run = (role: string, json: Record<string, unknown>) => api(role, 'hub/apps/run', { json })
+
+function zipForm(extra: Record<string, string>, bytes: Uint8Array): FormData {
+  const form = new FormData()
+  for (const [k, v] of Object.entries(extra)) form.append(k, v)
+  form.append('fileAppZip', new Blob([bytes], { type: 'application/zip' }), 'Blocker.zip')
+  return form
+}
+
+const writes = (f: Fake) => f.calls.filter((c) => c.path.startsWith('apps/') && c.method === 'POST')
+
+beforeAll(async () => {
+  a = await fake()
+  b = await fake()
+  dataDir = mkdtempSync(path.join(tmpdir(), 'hub-write-data-'))
+  staticDir = mkdtempSync(path.join(tmpdir(), 'hub-write-static-'))
+  let users = addUser([], 'viewer', 'viewer', 'password-viewer')
+  users = addUser(users, 'operator', 'operator', 'password-operator')
+  users = addUser(users, 'apps-admin', 'apps-admin', 'password-apps-admin')
+  users = addUser(users, 'admin', 'admin', 'password-admin')
+  writeUsersFile(dataDir, users)
+})
+
+afterAll(() => {
+  hub?.close()
+  a.server.close()
+  b.server.close()
+  rmSync(dataDir, { recursive: true, force: true })
+  rmSync(staticDir, { recursive: true, force: true })
+})
+
+beforeEach(async () => {
+  reset()
+  rmSync(path.join(dataDir, 'apps'), { recursive: true, force: true })
+  rmSync(path.join(dataDir, 'audit.log'), { force: true })
+  await start()
+})
+
+afterEach(() => {
+  hub?.close()
+})
+
+describe('multipart parser', () => {
+  const boundary = '----testBoundary123'
+  const type = `multipart/form-data; boundary=${boundary}`
+
+  it('reads fields and a binary file, whatever bytes it holds', () => {
+    const binary = Buffer.alloc(256)
+    for (let i = 0; i < 256; i++) binary[i] = i
+    const tricky = Buffer.concat([binary, Buffer.from(`\r\n--${boundary.slice(0, 5)}\r\n\r\n--`), binary])
+    const body = Buffer.concat([
+      Buffer.from(`--${boundary}\r\nContent-Disposition: form-data; name="name"\r\n\r\nBlocker\r\n`),
+      Buffer.from(`--${boundary}\r\nContent-Disposition: form-data; name="fileAppZip"; filename="a b.zip"\r\nContent-Type: application/zip\r\n\r\n`),
+      tricky,
+      Buffer.from(`\r\n--${boundary}--\r\n`),
+    ])
+    const parsed = parseMultipart(body, type)
+    expect(parsed.fields).toEqual({ name: 'Blocker' })
+    expect(parsed.files).toHaveLength(1)
+    expect(parsed.files[0]).toMatchObject({ field: 'fileAppZip', filename: 'a b.zip', type: 'application/zip' })
+    expect(parsed.files[0].data.equals(tricky)).toBe(true)
+  })
+
+  it('reads a quoted boundary and an empty file', () => {
+    expect(boundaryOf('multipart/form-data; boundary="x y"')).toBe('x y')
+    const body = Buffer.from(`--q\r\nContent-Disposition: form-data; name="f"; filename="e.zip"\r\n\r\n\r\n--q--`)
+    expect(parseMultipart(body, 'multipart/form-data; boundary=q').files[0].data.length).toBe(0)
+  })
+
+  it('refuses what is not multipart', () => {
+    expect(() => parseMultipart(Buffer.from('x'), 'text/plain')).toThrow()
+    expect(() => parseMultipart(Buffer.from('nothing here'), type)).toThrow(/boundary/)
+    expect(() => parseMultipart(Buffer.from(`--${boundary}\r\nContent-Disposition: form-data; name="a"\r\n\r\nno end`), type)).toThrow(/terminated/)
+  })
+
+  it('names folders safely', () => {
+    expect(safeName('Blocker')).toBe('Blocker')
+    expect(safeName('Query Logs (Sqlite)')).toMatch(/^Query_Logs__Sqlite_-[0-9a-f]{6}$/)
+    expect(safeName('../../etc')).not.toContain('/')
+    expect(safeName('..')).not.toMatch(/^\./)
+  })
+})
+
+describe('roles', () => {
+  it('refuses viewer and operator on every write and audits it', async () => {
+    for (const role of ['viewer', 'operator']) {
+      const r = await run(role, { action: 'uninstall', name: 'Blocker', targets: ['a'] })
+      expect(r.code).toBe(403)
+      expect(r.body).toEqual({ status: 'error', errorMessage: 'Your role does not allow this.' })
+      expect((await api(role, 'apps/uninstall', { params: { name: 'Blocker', node: 'a' } })).code).toBe(403)
+      expect((await api(role, 'hub/apps/upload', { body: zipForm({ action: 'install', name: 'X', targets: 'a' }, new Uint8Array([1])) })).code).toBe(403)
+    }
+    expect(writes(a)).toHaveLength(0)
+    const entries = (await api('admin', 'hub/audit/list', { params: { limit: '100' } })).body.response.entries as Record<string, string>[]
+    expect(entries.filter((e) => e.action === 'denied-write' && e.user === 'viewer').length).toBeGreaterThanOrEqual(3)
+    expect(entries.some((e) => e.action === 'denied-write' && e.user === 'operator' && e.target === 'hub/apps/run')).toBe(true)
+  })
+
+  it('lets apps-admin and admin write', async () => {
+    for (const role of ['apps-admin', 'admin']) {
+      b.apps = a.apps = { Blocker: { version: '1.0', config: null } }
+      const r = await run(role, { action: 'update-store', name: 'Blocker', targets: ['a'] })
+      expect(r.body.status).toBe('ok')
+      expect(r.body.response.results).toEqual([{ server: 'a', ok: true, message: 'Updated from the store.' }])
+    }
+  })
+
+  it('does not let a write endpoint be called with GET', async () => {
+    expect((await api('admin', 'hub/apps/run', { method: 'GET', params: { action: 'uninstall', name: 'Blocker', targets: 'a' } })).code).toBe(405)
+    expect(writes(a)).toHaveLength(0)
+  })
+
+  it('tells the session who may write what', async () => {
+    const viewer = (await api('viewer', 'user/session/get')).body.info
+    const admin = (await api('apps-admin', 'user/session/get')).body.info
+    expect(viewer.permissions.Apps).toEqual({ canView: true, canModify: false, canDelete: false })
+    expect(admin.permissions.Apps).toEqual({ canView: true, canModify: true, canDelete: true })
+    expect(admin.permissions.Zones.canModify).toBe(false)
+    expect(admin.hubWritable).toEqual(['a', 'b'])
+    await start({ B_ADMIN_TOKEN: '' })
+    expect((await api('viewer', 'user/session/get')).body.info.hubWritable).toEqual(['a'])
+  })
+})
+
+describe('hub/apps/run', () => {
+  it('uses only the admin token and the URL of the target own store', async () => {
+    a.store = STORE('2.0')
+    b.store = [{ ...STORE('2.5')[0], url: 'https://other.example/b.zip' }]
+    const r = await run('admin', { action: 'update-store', name: 'Blocker', targets: ['a', 'b'] })
+    expect(r.body.response.results.map((x: any) => x.ok)).toEqual([true, true])
+    const [wa] = writes(a)
+    const [wb] = writes(b)
+    expect(wa.token).toBe('admin-a')
+    expect(wa.query.get('url')).toBe('https://store.example/Blocker-v2.0.zip')
+    expect(wb.token).toBe('admin-b')
+    expect(wb.query.get('url')).toBe('https://other.example/b.zip')
+    expect(a.calls.filter((c) => c.token === 'read-a').every((c) => !c.path.includes('downloadAnd'))).toBe(true)
+  })
+
+  it('refuses a url that is not from the store listing, and an app the store does not have', async () => {
+    const r = await run('admin', { action: 'install-store', name: 'Blocker', url: 'https://evil.example/x.zip', targets: ['a'] })
+    expect(r.code).toBe(400)
+    expect(r.body.errorMessage).toMatch(/store listing/)
+    expect(writes(a)).toHaveLength(0)
+
+    const same = await run('admin', { action: 'update-store', name: 'Blocker', url: STORE('2.0')[0].url, targets: ['a'] })
+    expect(same.body.status).toBe('ok')
+
+    const missing = await run('admin', { action: 'install-store', name: 'Nope', targets: ['a'] })
+    expect(missing.body.response.results[0]).toMatchObject({ server: 'a', ok: false })
+    expect(missing.body.response.results[0].message).toMatch(/not in this server/)
+  })
+
+  it('refuses unknown servers, no targets and bad actions', async () => {
+    expect((await run('admin', { action: 'update-store', name: 'Blocker', targets: ['zzz'] })).code).toBe(400)
+    expect((await run('admin', { action: 'update-store', name: 'Blocker', targets: [] })).code).toBe(400)
+    expect((await run('admin', { action: 'format', name: 'Blocker', targets: ['a'] })).code).toBe(400)
+    expect((await run('admin', { action: 'uninstall', name: '', targets: ['a'] })).code).toBe(400)
+  })
+
+  it('accepts a form body with comma separated targets', async () => {
+    const r = await api('admin', 'hub/apps/run', {
+      body: new URLSearchParams({ action: 'update-store', name: 'Blocker', targets: 'a,b' }).toString(),
+      headers: { 'content-type': 'application/x-www-form-urlencoded' },
+    })
+    expect(r.body.response.results).toHaveLength(2)
+  })
+
+  it('refuses a server without an admin token, without calling it', async () => {
+    await start({ B_ADMIN_TOKEN: '' })
+    const r = await run('admin', { action: 'update-store', name: 'Blocker', targets: ['a', 'b'] })
+    const results = r.body.response.results
+    expect(results.find((x: any) => x.server === 'a').ok).toBe(true)
+    expect(results.find((x: any) => x.server === 'b')).toEqual({ server: 'b', ok: false, message: 'no admin token configured' })
+    expect(writes(b)).toHaveLength(0)
+    expect(b.calls.every((c) => c.token === 'read-b')).toBe(true)
+    const m = await api('admin', 'hub/apps/matrix')
+    expect(m.body.response.servers).toEqual([{ id: 'a', ok: true, writable: true }, { id: 'b', ok: true, writable: false }])
+  })
+
+  it('reports each target on its own when one fails, and audits every result', async () => {
+    b.failWrites = true
+    const r = await run('apps-admin', { action: 'update-store', name: 'Blocker', targets: ['a', 'b'] })
+    expect(r.body.response.results).toEqual([
+      { server: 'a', ok: true, message: 'Updated from the store.' },
+      { server: 'b', ok: false, message: 'Disk full.' },
+    ])
+    expect(a.apps.Blocker.version).toBe('2.0')
+    expect(b.apps.Blocker.version).toBe('1.0')
+    const entries = (await api('admin', 'hub/audit/list', { params: { limit: '20' } })).body.response.entries as Record<string, string>[]
+    const mine = entries.filter((e) => e.action === 'apps.update-store')
+    expect(mine).toHaveLength(2)
+    expect(mine.find((e) => e.server === 'a')).toMatchObject({ user: 'apps-admin', role: 'apps-admin', target: 'Blocker', outcome: 'ok' })
+    expect(mine.find((e) => e.server === 'b')).toMatchObject({ outcome: 'failed', detail: 'Disk full.' })
+    const raw = readFileSync(path.join(dataDir, 'audit.log'), 'utf8')
+    for (const secret of ['admin-a', 'admin-b', 'read-a', 'password-apps-admin', '"secret":"cfg"']) expect(raw).not.toContain(secret)
+  })
+
+  it('shows the new version in the matrix after an update (the cache is dropped)', async () => {
+    await start({ HUB_CACHE_SECONDS: '60' })
+    const before = (await api('viewer', 'hub/apps/matrix')).body.response.apps[0]
+    expect(before.servers.a.version).toBe('1.0')
+    await run('admin', { action: 'update-store', name: 'Blocker', targets: ['a'] })
+    const after = (await api('viewer', 'hub/apps/matrix')).body.response.apps[0]
+    expect(after.servers.a.version).toBe('2.0')
+    expect(after.servers.b.version).toBe('1.0')
+    expect(after.versions).toBe('different')
+  })
+})
+
+describe('uninstall and its history', () => {
+  const history = () => path.join(dataDir, 'apps', 'Blocker', 'history')
+
+  it('saves the config first, then uninstalls', async () => {
+    const r = await run('admin', { action: 'uninstall', name: 'Blocker', targets: ['a', 'b'] })
+    expect(r.body.response.results.map((x: any) => x.ok)).toEqual([true, true])
+    expect(a.apps.Blocker).toBeUndefined()
+    const files = readdirSync(history()).sort()
+    expect(files).toHaveLength(2)
+    expect(files.some((f) => /^\d{4}-\d\d-\d\dT\d\d-\d\d-\d\d\.\d{3}Z__a__before-uninstall\.txt$/.test(f))).toBe(true)
+    expect(readFileSync(path.join(history(), files[0]), 'utf8')).toBe('{"secret":"cfg"}')
+    expect(readdirSync(history()).some((f) => f.endsWith('.tmp'))).toBe(false)
+  })
+
+  it('keeps at most 50 files per app, dropping the oldest', async () => {
+    mkdirSync(history(), { recursive: true })
+    for (let i = 0; i < 55; i++) writeFileSync(path.join(history(), `2020-01-01T00-00-${String(i).padStart(2, '0')}.000Z__a__before-uninstall.txt`), 'old')
+    await run('admin', { action: 'uninstall', name: 'Blocker', targets: ['a'] })
+    const files = readdirSync(history()).sort()
+    expect(files).toHaveLength(50)
+    expect(files.some((f) => f.startsWith('2020-01-01T00-00-00.'))).toBe(false)
+    expect(files.some((f) => f.includes('__a__') && f.startsWith('20') && !f.startsWith('2020-'))).toBe(true)
+  })
+
+  it('does not uninstall when the config cannot be saved', async () => {
+    a.denyConfig = true
+    const r = await run('admin', { action: 'uninstall', name: 'Blocker', targets: ['a'] })
+    expect(r.body.response.results[0].ok).toBe(false)
+    expect(r.body.response.results[0].message).toMatch(/nothing was uninstalled/)
+    expect(a.apps.Blocker).toBeDefined()
+    expect(writes(a)).toHaveLength(0)
+  })
+
+  it('keeps an odd app name inside the data folder', async () => {
+    a.apps['../../evil'] = { version: '1', config: 'x' }
+    await run('admin', { action: 'uninstall', name: '../../evil', targets: ['a'] })
+    expect(existsSync(path.join(dataDir, '..', 'evil'))).toBe(false)
+    const dirs = readdirSync(path.join(dataDir, 'apps'))
+    expect(dirs.every((d) => d !== '.' && d !== '..' && !/[\/]/.test(d) && !d.includes('..'))).toBe(true)
+  })
+})
+
+describe('hub/apps/upload', () => {
+  const bytes = new Uint8Array(70_000).map((_, i) => (i * 7 + 3) % 256)
+
+  it('forwards the same zip to every target, byte for byte', async () => {
+    b.apps = {}
+    a.apps = {}
+    const r = await api('apps-admin', 'hub/apps/upload', { body: zipForm({ action: 'install', name: 'Blocker', targets: 'a,b' }, bytes) })
+    expect(r.body.response.results).toEqual([
+      { server: 'a', ok: true, message: 'Installed from the file.' },
+      { server: 'b', ok: true, message: 'Installed from the file.' },
+    ])
+    for (const [f, admin] of [[a, 'admin-a'], [b, 'admin-b']] as const) {
+      const [w] = writes(f)
+      expect(w.path).toBe('apps/install')
+      expect(w.token).toBe(admin)
+      expect(w.query.get('name')).toBe('Blocker')
+      const parsed = parseMultipart(w.body, w.contentType)
+      expect(Buffer.from(parsed.files[0].data).equals(Buffer.from(bytes))).toBe(true)
+    }
+    const audit = (await api('admin', 'hub/audit/list', { params: { limit: '10' } })).body.response.entries as Record<string, string>[]
+    expect(audit.filter((e) => e.action === 'apps.install-file' && e.outcome === 'ok')).toHaveLength(2)
+  })
+
+  it('updates, and reports an app that is not there as that target failure', async () => {
+    delete b.apps.Blocker
+    const r = await api('admin', 'hub/apps/upload', { body: zipForm({ action: 'update', name: 'Blocker', targets: 'a,b' }, bytes) })
+    expect(writes(a)[0].path).toBe('apps/update')
+    expect(r.body.response.results.map((x: any) => x.ok)).toEqual([true, true])
+  })
+
+  it('refuses a missing file, a bad action and a body above 20 MB', async () => {
+    const noFile = new FormData()
+    noFile.append('action', 'install')
+    noFile.append('name', 'X')
+    noFile.append('targets', 'a')
+    expect((await api('admin', 'hub/apps/upload', { body: noFile })).code).toBe(400)
+    expect((await api('admin', 'hub/apps/upload', { body: zipForm({ action: 'wipe', name: 'X', targets: 'a' }, bytes) })).code).toBe(400)
+
+    const big = new Uint8Array(21 * 1024 * 1024)
+    const r = await api('admin', 'hub/apps/upload', { body: zipForm({ action: 'install', name: 'X', targets: 'a' }, big) })
+    expect(r.code).toBe(413)
+    expect(r.body.errorMessage).toMatch(/20 MB/)
+    expect(writes(a)).toHaveLength(0)
+  })
+
+  it('does not read a big body from someone who is not allowed to upload', async () => {
+    const r = await api('viewer', 'hub/apps/upload', { body: zipForm({ action: 'install', name: 'X', targets: 'a' }, bytes) })
+    expect(r.code).toBe(403)
+  })
+})
+
+describe('single-server writes', () => {
+  it('forwards with the admin token', async () => {
+    const r = await api('apps-admin', 'apps/downloadAndUpdate', { params: { node: 'a', name: 'Blocker', url: STORE('2.0')[0].url } })
+    expect(r.body.status).toBe('ok')
+    const [w] = writes(a)
+    expect(w).toMatchObject({ path: 'apps/downloadAndUpdate', token: 'admin-a' })
+    expect(w.query.get('node')).toBeNull()
+    expect(writes(b)).toHaveLength(0)
+    expect(a.apps.Blocker.version).toBe('2.0')
+  })
+
+  it('works with a form POST too, and uninstalls with the history', async () => {
+    const r = await api('admin', 'apps/uninstall', {
+      body: new URLSearchParams({ node: 'b', name: 'Blocker' }).toString(),
+      headers: { 'content-type': 'application/x-www-form-urlencoded' },
+    })
+    expect(r.body.status).toBe('ok')
+    expect(b.apps.Blocker).toBeUndefined()
+    expect(readdirSync(path.join(dataDir, 'apps', 'Blocker', 'history'))).toHaveLength(1)
+  })
+
+  it('is refused without a node or with the aggregate', async () => {
+    for (const params of [{ name: 'Blocker' } as Record<string, string>, { name: 'Blocker', node: 'cluster' }, { name: 'Blocker', node: 'zzz' }]) {
+      const r = await api('admin', 'apps/uninstall', { params })
+      expect(r.code).toBe(400)
+    }
+    expect(writes(a)).toHaveLength(0)
+    expect(writes(b)).toHaveLength(0)
+  })
+
+  it('does not forward a download url the store did not offer', async () => {
+    const r = await api('admin', 'apps/downloadAndInstall', { params: { node: 'a', name: 'Blocker', url: 'https://evil.example/x.zip' } })
+    expect(r.body.status).toBe('error')
+    expect(r.body.errorMessage).toMatch(/store listing/)
+    expect(writes(a)).toHaveLength(0)
+  })
+
+  it('reports a server without an admin token', async () => {
+    await start({ A_ADMIN_TOKEN: '' })
+    const r = await api('admin', 'apps/uninstall', { params: { node: 'a', name: 'Blocker' } })
+    expect(r.body).toEqual({ status: 'error', errorMessage: 'a: no admin token configured' })
+    expect(a.calls.filter((c) => c.method === 'POST')).toHaveLength(0)
+  })
+
+  it('forwards a multipart install to the node', async () => {
+    delete a.apps.Blocker
+    const bytes = new Uint8Array([80, 75, 3, 4, 13, 10, 45, 45, 0, 255])
+    const r = await api('admin', 'apps/install', { params: { node: 'a', name: 'Blocker' }, body: zipForm({}, bytes) })
+    expect(r.body.status).toBe('ok')
+    const [w] = writes(a)
+    expect(w.token).toBe('admin-a')
+    expect(Buffer.from(parseMultipart(w.body, w.contentType).files[0].data).equals(Buffer.from(bytes))).toBe(true)
+  })
+
+  it('keeps config get and set refused for now', async () => {
+    for (const p of ['apps/config/get', 'apps/config/set']) {
+      expect((await api('admin', p, { params: { node: 'a', name: 'Blocker' } })).code).toBe(403)
+    }
+  })
+
+  it('lists the store with the servers each app is installed on', async () => {
+    delete b.apps.Blocker
+    const r = await api('viewer', 'apps/listStoreApps')
+    expect(r.body.response.storeApps[0]).toMatchObject({ name: 'Blocker', installedOn: ['a'] })
+  })
+})

hub/apps-write.ts

@@ -0,0 +1,185 @@
+/*
+App writes: install, update and uninstall on one or several servers. The first part of the
+hub that changes anything, so the rules are strict:
+
+  * only a server's ADMIN token writes (`<ID>_ADMIN_TOKEN`); a server without one is refused;
+  * a store download URL is never taken from the client: it is looked up in the target
+    server's own `apps/listStoreApps`, and a URL the client sends must match it;
+  * before an uninstall the app's config is saved to `${HUB_DATA}/apps/<app>/history/`, and
+    the uninstall is refused if that cannot be done;
+  * every target's result is audited (the caller does that with `Audit`).
+*/
+import { createHash } from 'node:crypto'
+import { mkdirSync, readdirSync, renameSync, rmSync, writeFileSync } from 'node:fs'
+import path from 'node:path'
+import type { HubConfig, HubServer } from './config.ts'
+import type { UploadedFile } from './multipart.ts'
+import { Upstream } from './upstream.ts'
+
+export type StoreAction = 'install-store' | 'update-store' | 'uninstall'
+export type FileAction = 'install-file' | 'update-file'
+export type AppAction = StoreAction | FileAction
+
+export interface TargetResult {
+  server: string
+  ok: boolean
+  message?: string
+}
+
+export const KEEP_HISTORY = 50
+
+const API: Record<AppAction, string> = {
+  'install-store': 'apps/downloadAndInstall',
+  'update-store': 'apps/downloadAndUpdate',
+  uninstall: 'apps/uninstall',
+  'install-file': 'apps/install',
+  'update-file': 'apps/update',
+}
+
+const DONE: Record<AppAction, string> = {
+  'install-store': 'Installed from the store.',
+  'update-store': 'Updated from the store.',
+  uninstall: 'Uninstalled.',
+  'install-file': 'Installed from the file.',
+  'update-file': 'Updated from the file.',
+}
+
+/** The audit/api action for an upstream write path, or null when it is not an app write. */
+export function actionForPath(apiPath: string): AppAction | null {
+  for (const [action, p] of Object.entries(API)) if (p === apiPath) return action as AppAction
+  return null
+}
+
+/* A folder name for an app: letters, digits and . _ - only, with a short hash when anything had to change. */
+export function safeName(name: string): string {
+  const cleaned = name.replace(/[^A-Za-z0-9._-]/g, '_').replace(/\.{2,}/g, '_').replace(/^\.+/, '_').slice(0, 80)
+  if (cleaned === name && cleaned !== '') return cleaned
+  return `${cleaned === '' ? '_' : cleaned}-${createHash('sha256').update(name).digest('hex').slice(0, 6)}`
+}
+
+const reason = (e: unknown): string => (e instanceof Error ? e.message : 'failed')
+
+export class AppsWriter {
+  #config: HubConfig
+  #upstream: Upstream
+  #now: () => number
+
+  constructor(config: HubConfig, upstream: Upstream, now: () => number = Date.now) {
+    this.#config = config
+    this.#upstream = upstream
+    this.#now = now
+  }
+
+  /** The ids of the servers that have an admin token. */
+  writableIds(): string[] {
+    return this.#config.servers.filter((s) => s.adminToken !== undefined).map((s) => s.id)
+  }
+
+  /* The URL the server's own store gives for this app, or null. */
+  async storeUrl(server: HubServer, name: string): Promise<string | null> {
+    const response = (await this.#upstream.call(server, 'apps/listStoreApps')) as { storeApps?: { name: string; url?: string }[] } | null
+    const found = (response?.storeApps ?? []).find((a) => a.name === name)
+    return typeof found?.url === 'string' && found.url !== '' ? found.url : null
+  }
+
+  /* Saves the app's current config next to the hub's data; throws if it cannot. */
+  async saveHistory(server: HubServer, name: string): Promise<string> {
+    if (server.adminToken === undefined) throw new Error('no admin token configured')
+    this.#upstream.invalidate(server.id)
+    const response = (await this.#upstream.call(server, 'apps/config/get', { name }, server.adminToken)) as { config?: unknown } | null
+    const text = typeof response?.config === 'string' ? response.config : ''
+    const dir = path.join(this.#config.dataDir, 'apps', safeName(name), 'history')
+    mkdirSync(dir, { recursive: true })
+    const stamp = new Date(this.#now()).toISOString().replace(/:/g, '-')
+    const file = path.join(dir, `${stamp}__${safeName(server.id)}__before-uninstall.txt`)
+    const temp = `${file}.${process.pid}.tmp`
+    writeFileSync(temp, text, { mode: 0o600 })
+    renameSync(temp, file)
+    // Oldest first by name (the name starts with the time); keep the newest KEEP_HISTORY.
+    const files = readdirSync(dir).filter((f) => f.endsWith('.txt')).sort()
+    for (const old of files.slice(0, Math.max(0, files.length - KEEP_HISTORY))) rmSync(path.join(dir, old), { force: true })
+    return file
+  }
+
+  /*
+  One write on one server. `url` must already be checked against the store; `file` is the zip
+  for the two file actions. Returns the server's response, or throws.
+  */
+  async perform(server: HubServer, action: AppAction, name: string, url?: string, file?: UploadedFile): Promise<unknown> {
+    if (server.adminToken === undefined) throw new Error('no admin token configured')
+    switch (action) {
+      case 'install-store':
+      case 'update-store':
+        return await this.#upstream.write(server, API[action], { name, url: url ?? '' })
+      case 'uninstall':
+        try {
+          await this.saveHistory(server, name)
+        } catch (e) {
+          throw new Error(`the config could not be saved before the uninstall (${reason(e)}); nothing was uninstalled`)
+        }
+        return await this.#upstream.write(server, API[action], { name })
+      case 'install-file':
+      case 'update-file': {
+        if (file === undefined) throw new Error('no zip file')
+        const form = new FormData()
+        form.append('fileAppZip', new Blob([file.data], { type: file.type }), file.filename)
+        return await this.#upstream.write(server, API[action], { name }, form)
+      }
+    }
+  }
+
+  /*
+  The same write on several servers, in parallel. A store action first resolves each target's
+  URL from its own store; `suppliedUrl` (if the client sent one) has to match every one of them.
+  Returns `{ error }` when the request itself is refused, else one result per target.
+  */
+  async runMany(
+    action: AppAction,
+    name: string,
+    targets: string[],
+    options: { suppliedUrl?: string; file?: UploadedFile } = {},
+  ): Promise<{ error: string } | { results: TargetResult[] }> {
+    const known = new Map(this.#config.servers.map((s) => [s.id, s]))
+    const unique = [...new Set(targets)]
+    if (name === '' || name.length > 200) return { error: 'An app name is required.' }
+    if (unique.length === 0) return { error: 'Choose at least one server.' }
+    const unknown = unique.filter((t) => !known.has(t))
+    if (unknown.length > 0) return { error: `Unknown server: ${unknown.join(', ')}.` }
+
+    const urls = new Map<string, string>()
+    const early = new Map<string, string>()
+    if (action === 'install-store' || action === 'update-store') {
+      await Promise.all(
+        unique.map(async (id) => {
+          const server = known.get(id) as HubServer
+          if (server.adminToken === undefined) return early.set(id, 'no admin token configured')
+          try {
+            const url = await this.storeUrl(server, name)
+            if (url === null) early.set(id, `"${name}" is not in this server's store listing`)
+            else urls.set(id, url)
+          } catch (e) {
+            early.set(id, `the store listing could not be read (${reason(e)})`)
+          }
+        }),
+      )
+      if (options.suppliedUrl !== undefined && options.suppliedUrl !== '' && [...urls.values()].some((u) => u !== options.suppliedUrl)) {
+        return { error: 'The url is not from the store listing; the hub only installs what a server offers.' }
+      }
+    }
+
+    const results = await Promise.all(
+      unique.map(async (id): Promise<TargetResult> => {
+        const server = known.get(id) as HubServer
+        const refused = early.get(id) ?? (server.adminToken === undefined ? 'no admin token configured' : undefined)
+        if (refused !== undefined) return { server: id, ok: false, message: refused }
+        try {
+          await this.perform(server, action, name, urls.get(id), options.file)
+          return { server: id, ok: true, message: DONE[action] }
+        } catch (e) {
+          return { server: id, ok: false, message: reason(e) }
+        }
+      }),
+    )
+    return { results }
+  }
+}

hub/multipart.ts

@@ -0,0 +1,91 @@
+/*
+A small multipart/form-data reader for the app zip uploads, and a body reader with a size
+limit. No dependencies; the whole body stays in memory (the limit is 20 MB).
+*/
+import type http from 'node:http'
+
+export const MAX_UPLOAD = 20 * 1024 * 1024
+
+export class TooLarge extends Error {
+  constructor(limit: number) {
+    super(`The upload is larger than ${Math.floor(limit / (1024 * 1024))} MB.`)
+    this.name = 'TooLarge'
+  }
+}
+
+/*
+Reads the request body, counting bytes as they arrive; throws `TooLarge` above `limit`.
+Past the limit nothing more is kept, but the rest is still read (up to DRAIN more bytes), so the
+client gets its 413 instead of a reset connection.
+*/
+const DRAIN = 32 * 1024 * 1024
+
+export async function readLimited(req: http.IncomingMessage, limit: number = MAX_UPLOAD): Promise<Buffer> {
+  const declared = Number(req.headers['content-length'])
+  if (Number.isFinite(declared) && declared > limit + DRAIN) throw new TooLarge(limit)
+  const chunks: Buffer[] = []
+  let size = 0
+  for await (const chunk of req) {
+    size += (chunk as Buffer).length
+    if (size > limit + DRAIN) throw new TooLarge(limit)
+    if (size <= limit) chunks.push(chunk as Buffer)
+  }
+  if (size > limit) throw new TooLarge(limit)
+  return Buffer.concat(chunks)
+}
+
+export interface UploadedFile {
+  field: string
+  filename: string
+  type: string
+  data: Buffer
+}
+
+export interface Multipart {
+  fields: Record<string, string>
+  files: UploadedFile[]
+}
+
+export function boundaryOf(contentType: string): string | null {
+  const match = /boundary=(?:"([^"]+)"|([^;\s]+))/i.exec(contentType)
+  return match === null ? null : (match[1] ?? match[2])
+}
+
+const CRLF2 = Buffer.from('\r\n\r\n')
+
+/* Throws on anything that is not a well-formed multipart body. */
+export function parseMultipart(body: Buffer, contentType: string): Multipart {
+  const boundary = boundaryOf(contentType)
+  if (boundary === null || !/^multipart\/form-data/i.test(contentType)) throw new Error('Expected multipart/form-data with a boundary.')
+  const delimiter = Buffer.from(`--${boundary}`)
+  const result: Multipart = { fields: {}, files: [] }
+
+  let at = body.indexOf(delimiter)
+  if (at < 0) throw new Error('The multipart body has no boundary.')
+  for (;;) {
+    at += delimiter.length
+    // "--" after a delimiter closes the body.
+    if (body[at] === 0x2d && body[at + 1] === 0x2d) return result
+    if (body[at] !== 0x0d || body[at + 1] !== 0x0a) throw new Error('Malformed multipart body.')
+    at += 2
+    const headerEnd = body.indexOf(CRLF2, at)
+    if (headerEnd < 0) throw new Error('Malformed multipart part.')
+    const headers = body.subarray(at, headerEnd).toString('utf8')
+    const start = headerEnd + CRLF2.length
+    // The part ends at the next CRLF + delimiter, so binary content can hold anything else.
+    const next = body.indexOf(Buffer.concat([Buffer.from('\r\n'), delimiter]), start)
+    if (next < 0) throw new Error('The multipart body is not terminated.')
+    const data = body.subarray(start, next)
+
+    const disposition = /^content-disposition:\s*form-data;(.*)$/im.exec(headers)?.[1] ?? ''
+    const name = /\bname="([^"]*)"/i.exec(disposition)?.[1]
+    const filename = /\bfilename="([^"]*)"/i.exec(disposition)?.[1]
+    if (name === undefined) throw new Error('A multipart part has no name.')
+    if (filename === undefined) result.fields[name] = data.toString('utf8')
+    else {
+      const type = /^content-type:\s*(.+)$/im.exec(headers)?.[1].trim() ?? 'application/octet-stream'
+      result.files.push({ field: name, filename, type, data: Buffer.from(data) })
+    }
+    at = next + 2
+  }
+}

hub/routes.ts

@@ -1,7 +1,7 @@
 /*
 What each `/api/<path>` does and who may call it. `kind` says what it touches:
 `read` (no change), `write-one` (changes one server) and `write-all` (changes every server).
-A path that is not listed is refused. Today every entry is a read.
+A path that is not listed is refused. The app writes need `apps.write`.
 
 Not in the table: `status`, `user/login`, `user/session/get` and `user/logout`, which the
 server answers itself before the table is consulted.
@@ -16,6 +16,8 @@ export interface Route {
 }
 
 const read = (capability: Capability = 'read'): Route => ({ kind: 'read', capability })
+const writeOne = (capability: Capability): Route => ({ kind: 'write-one', capability })
+const writeAll = (capability: Capability): Route => ({ kind: 'write-all', capability })
 
 export const ROUTES: Record<string, Route> = {
   'dashboard/stats/get': read(),
@@ -30,6 +32,15 @@ export const ROUTES: Record<string, Route> = {
   // The hub's own namespace; never forwarded to a server.
   'hub/audit/list': read('audit.read'),
   'hub/apps/matrix': read(),
+  'apps/listStoreApps': read(),
+  // Writes. They go to a server's ADMIN token only; see apps-write.ts. apps/config/set waits for a later stage.
+  'apps/downloadAndInstall': writeOne('apps.write'),
+  'apps/downloadAndUpdate': writeOne('apps.write'),
+  'apps/uninstall': writeOne('apps.write'),
+  'apps/install': writeOne('apps.write'),
+  'apps/update': writeOne('apps.write'),
+  'hub/apps/run': writeAll('apps.write'),
+  'hub/apps/upload': writeAll('apps.write'),
 }
 
 export function routeFor(apiPath: string): Route | undefined {

hub/server.ts

@@ -22,6 +22,8 @@ import { LoginLimiter, lockedMessage } from './ratelimit.ts'
 import { Audit } from './audit.ts'
 import { routeFor } from './routes.ts'
 import { buildMatrix } from './apps-matrix.ts'
+import { AppsWriter, actionForPath, type FileAction, type TargetResult } from './apps-write.ts'
+import { TooLarge, parseMultipart, readLimited, type UploadedFile } from './multipart.ts'
 import {
   mergeQueryLogs,
   mergeSettled,
@@ -80,10 +82,30 @@ async function readForm(req: http.IncomingMessage): Promise<URLSearchParams> {
     chunks.push(chunk as Buffer)
   }
   const type = String(req.headers['content-type'] ?? '')
-  if (size > MAX_BODY || !type.includes('application/x-www-form-urlencoded')) return new URLSearchParams()
-  return new URLSearchParams(Buffer.concat(chunks).toString('utf8'))
+  if (size > MAX_BODY) return new URLSearchParams()
+  const text = Buffer.concat(chunks).toString('utf8')
+  if (type.includes('application/x-www-form-urlencoded')) return new URLSearchParams(text)
+  if (type.includes('application/json')) {
+    // A flat object; an array becomes a comma-separated list, as the form fields are.
+    const out = new URLSearchParams()
+    try {
+      const data = JSON.parse(text) as Record<string, unknown>
+      for (const [k, v] of Object.entries(data)) {
+        if (Array.isArray(v)) out.set(k, v.map(String).join(','))
+        else if (typeof v === 'string' || typeof v === 'number' || typeof v === 'boolean') out.set(k, String(v))
+      }
+    } catch {
+      // Not JSON: no parameters.
+    }
+    return out
+  }
+  return new URLSearchParams()
 }
 
+const isMultipart = (req: http.IncomingMessage): boolean => String(req.headers['content-type'] ?? '').toLowerCase().startsWith('multipart/form-data')
+
+const csv = (value: string | undefined): string[] => (value ?? '').split(',').map((v) => v.trim()).filter((v) => v !== '')
+
 function bearer(req: http.IncomingMessage): string | null {
   const header = String(req.headers.authorization ?? '')
   return header.startsWith('Bearer ') ? header.slice(7).trim() : null
@@ -108,6 +130,7 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
   const sessions = new Sessions(config, users)
   const limiter = new LoginLimiter()
   const audit = new Audit(config.dataDir)
+  const writer = new AppsWriter(config, upstream)
   const byId = new Map(config.servers.map((s) => [s.id, s]))
 
   /* Ask every server; the ones that fail become warnings (see mergeSettled). */
@@ -162,6 +185,16 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
           const pages = data.map((page, i) => ({ node: answered[i], page }))
           return ok(mergeQueryLogs(pages, { pageNumber, entriesPerPage, descending: params.descendingOrder !== 'false' }), warnings)
         }
+        case 'apps/listStoreApps': {
+          // The first answering server's listing, each app tagged with the servers it is installed on.
+          const { data, warnings } = mergeSettled(await everywhere<{ storeApps?: { name: string }[] }>(apiPath, params))
+          const installedOn = new Map<string, string[]>()
+          for (const r of await everywhere<{ apps?: { name: string }[] }>('apps/list', {})) {
+            if (r.ok) for (const a of r.data.apps ?? []) installedOn.set(a.name, [...(installedOn.get(a.name) ?? []), r.node])
+          }
+          const storeApps = (data[0].storeApps ?? []).map((a) => ({ ...a, installedOn: installedOn.get(a.name) ?? [] }))
+          return ok({ ...data[0], storeApps }, warnings)
+        }
         case 'apps/list': {
           const { data, warnings } = mergeSettled(await everywhere<{ apps?: { name: string }[] }>(apiPath, params))
           const seen = new Map<string, { name: string }>()
@@ -178,8 +211,131 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
     }
   }
 
+  type Who = { user: string; role: string }
+
+  /* One audit line per target: who, which server, which app, what happened. */
+  function auditResults(who: Who, ip: string, action: string, name: string, results: TargetResult[]): void {
+    for (const r of results) {
+      audit.log({
+        user: who.user, role: who.role, ip, action: `apps.${action}`, server: r.server, target: name,
+        outcome: r.ok ? 'ok' : 'failed', detail: r.message,
+      })
+    }
+  }
+
+  async function uploadOf(req: http.IncomingMessage, res: http.ServerResponse, params: Params): Promise<UploadedFile | null> {
+    let parsed
+    try {
+      parsed = parseMultipart(await readLimited(req), String(req.headers['content-type'] ?? ''))
+    } catch (e) {
+      if (e instanceof TooLarge) {
+        res.setHeader('connection', 'close')
+        send(res, 413, fail(e.message))
+      } else send(res, 400, fail('The upload could not be read.'))
+      return null
+    }
+    Object.assign(params, parsed.fields)
+    const file = parsed.files[0]
+    if (file === undefined || file.data.length === 0) {
+      send(res, 400, fail('The zip file is missing.'))
+      return null
+    }
+    return file
+  }
+
+  /* The write endpoints. The role has already been checked against the route's capability. */
+  async function write(req: http.IncomingMessage, res: http.ServerResponse, apiPath: string, kind: string, params: Params, who: Who, ip: string): Promise<void> {
+    if (kind === 'write-all') {
+      if (req.method !== 'POST') {
+        send(res, 405, fail('Use POST.'))
+        return
+      }
+      if (apiPath === 'hub/apps/run') {
+        const action = params.action
+        if (action !== 'install-store' && action !== 'update-store' && action !== 'uninstall') {
+          send(res, 400, fail('action must be install-store, update-store or uninstall.'))
+          return
+        }
+        const outcome = await writer.runMany(action, params.name ?? '', csv(params.targets), { suppliedUrl: params.url })
+        if ('error' in outcome) {
+          audit.log({ user: who.user, role: who.role, ip, action: `apps.${action}`, target: params.name, outcome: 'refused', detail: outcome.error })
+          send(res, 400, fail(outcome.error))
+          return
+        }
+        auditResults(who, ip, action, params.name ?? '', outcome.results)
+        send(res, 200, ok({ results: outcome.results }))
+        return
+      }
+      if (apiPath === 'hub/apps/upload') {
+        if (!isMultipart(req)) {
+          send(res, 400, fail('Send multipart/form-data.'))
+          return
+        }
+        const file = await uploadOf(req, res, params)
+        if (file === null) return
+        if (params.action !== 'install' && params.action !== 'update') {
+          send(res, 400, fail('action must be install or update.'))
+          return
+        }
+        const action: FileAction = params.action === 'install' ? 'install-file' : 'update-file'
+        const outcome = await writer.runMany(action, params.name ?? '', csv(params.targets), { file })
+        if ('error' in outcome) {
+          audit.log({ user: who.user, role: who.role, ip, action: `apps.${action}`, target: params.name, outcome: 'refused', detail: outcome.error })
+          send(res, 400, fail(outcome.error))
+          return
+        }
+        auditResults(who, ip, action, params.name ?? '', outcome.results)
+        send(res, 200, ok({ results: outcome.results }))
+        return
+      }
+    }
+
+    // write-one: an upstream endpoint aimed at ONE server (`node=<id>`).
+    const action = actionForPath(apiPath)
+    const node = params.node ?? ''
+    const server = byId.get(node)
+    if (action === null) {
+      send(res, 403, fail('The hub is read-only.'))
+      return
+    }
+    if (server === undefined) {
+      send(res, 400, fail(node === '' || node === 'cluster' ? 'Choose one server: this change cannot be made on every server at once here.' : `Unknown node: ${node}`))
+      return
+    }
+    const name = params.name ?? ''
+    let file: UploadedFile | undefined
+    if (action === 'install-file' || action === 'update-file') {
+      if (!isMultipart(req)) {
+        send(res, 400, fail('Send multipart/form-data.'))
+        return
+      }
+      const uploaded = await uploadOf(req, res, params)
+      if (uploaded === null) return
+      file = uploaded
+    }
+    let result: TargetResult
+    let response: unknown = {}
+    try {
+      let url: string | undefined
+      if (action === 'install-store' || action === 'update-store') {
+        const fromStore = server.adminToken === undefined ? null : await writer.storeUrl(server, name).catch(() => null)
+        if (server.adminToken !== undefined && (fromStore === null || params.url !== fromStore)) {
+          throw new Error("the url is not from this server's store listing")
+        }
+        url = fromStore ?? undefined
+      }
+      response = await writer.perform(server, action, name, url, file)
+      result = { server: server.id, ok: true }
+    } catch (e) {
+      result = { server: server.id, ok: false, message: e instanceof Error ? e.message : 'failed' }
+    }
+    auditResults(who, ip, action, name, [result])
+    send(res, 200, result.ok ? ok(response) : fail(`${server.id}: ${result.message}`))
+  }
+
   async function api(req: http.IncomingMessage, res: http.ServerResponse, apiPath: string, url: URL): Promise<void> {
-    const form = await readForm(req)
+    // A multipart body (an app zip) is read later, once the caller is known to be allowed to send it.
+    const form = isMultipart(req) ? new URLSearchParams() : await readForm(req)
     const params: Params = {}
     for (const [k, v] of url.searchParams) params[k] = v
     for (const [k, v] of form) params[k] = v
@@ -249,6 +405,10 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
       send(res, 200, ok({ entries: audit.list(Number(params.limit ?? 200), params.before) }))
       return
     }
+    if (route.kind !== 'read') {
+      await write(req, res, apiPath, route.kind, params, who, ip)
+      return
+    }
     if (apiPath === 'hub/apps/matrix') {
       try {
         const { matrix, warnings } = await buildMatrix(config, upstream)

hub/session.ts

@@ -6,7 +6,7 @@ the console reads from `user/login` and `user/session/get` (flat, as the real se
 */
 import { randomBytes } from 'node:crypto'
 import { HUB_VERSION, type HubConfig } from './config.ts'
-import { signature, type Role, type UserRecord, type Users } from './users.ts'
+import { can, signature, type Role, type UserRecord, type Users } from './users.ts'
 
 export const IDLE_MS = 30 * 60_000
 
@@ -23,8 +23,8 @@ interface Session extends Identity {
 /*
 Every key `src/app/sections.ts` reads, spelled out: the console hides a section only
 when `canView === false`, so a missing key would leave it visible. The hub shows the
-Dashboard, the Logs and the Apps; About has no permission and is always there. The hub is
-read-only, so nothing can be modified or deleted.
+Dashboard, the Logs and the Apps; About has no permission and is always there. Only the Apps can be
+modified or deleted, and only by a role with `apps.write`.
 */
 const PERMISSION_KEYS = [
   'Dashboard', 'Zones', 'Cache', 'Allowed', 'Blocked', 'Apps', 'DnsClient', 'Settings', 'DhcpServer', 'Administration', 'Logs',
@@ -84,7 +84,10 @@ export class Sessions {
       uptimestamp: this.#started,
       dnsServerDomain: domain,
       permissions: Object.fromEntries(
-        PERMISSION_KEYS.map((key) => [key, { canView: VIEWABLE.has(key), canModify: false, canDelete: false }]),
+        PERMISSION_KEYS.map((key) => {
+          const writes = key === 'Apps' && can(role, 'apps.write')
+          return [key, { canView: VIEWABLE.has(key), canModify: writes, canDelete: writes }]
+        }),
       ),
       /* The console offers its node selector when the cluster is initialised. The
          first server plays the Primary (`primaryNodeName` looks for one). */
@@ -98,6 +101,8 @@ export class Sessions {
       dnssecValidation: true,
       hub: true,
       hubRole: role,
+      /* The servers the hub holds an admin token for: the only ones a write can reach. */
+      hubWritable: servers.filter((s) => s.adminToken !== undefined).map((s) => s.id),
     }
   }
 

hub/upstream.ts

@@ -28,6 +28,9 @@ interface Cached {
 
 const MAX_CACHED = 500
 
+/* An app download or upload can take a while. */
+const WRITE_TIMEOUT_MS = 60_000
+
 export class Upstream {
   #config: HubConfig
   #fetch: typeof fetch
@@ -47,20 +50,50 @@ export class Upstream {
     const hit = this.#cache.get(key)
     if (hit !== undefined && hit.until > this.#now()) return hit.response
 
-    const query = new URLSearchParams(sorted)
+    const response = await this.#exec(server, path, new URLSearchParams(sorted), token, { method: 'GET' }, this.#config.timeoutMs)
+    if (this.#config.cacheSeconds > 0) {
+      if (this.#cache.size >= MAX_CACHED) this.#prune()
+      this.#cache.set(key, { until: this.#now() + this.#config.cacheSeconds * 1000, response })
+    }
+    return response
+  }
+
+  /*
+  A write: POST with the server's ADMIN token and never the read one, never cached. Whatever
+  the outcome, what the hub remembers of this server is dropped afterwards, so the next read
+  shows the state the write left. `body` is a multipart upload (the app zip).
+  */
+  async write(server: HubServer, path: string, params: Params = {}, body?: FormData): Promise<unknown> {
+    if (server.adminToken === undefined) throw new UpstreamError('rejected', 'no admin token configured')
+    try {
+      const init: RequestInit = body === undefined ? { method: 'POST' } : { method: 'POST', body }
+      return await this.#exec(server, path, new URLSearchParams(params), server.adminToken, init, Math.max(this.#config.timeoutMs, WRITE_TIMEOUT_MS))
+    } finally {
+      this.invalidate(server.id)
+    }
+  }
+
+  /** Forget every cached answer of one server. */
+  invalidate(serverId: string): void {
+    const prefix = `${serverId}\n`
+    for (const key of this.#cache.keys()) if (key.startsWith(prefix)) this.#cache.delete(key)
+  }
+
+  async #exec(server: HubServer, path: string, query: URLSearchParams, token: string, init: RequestInit, timeoutMs: number): Promise<unknown> {
     query.set('token', token)
     const controller = new AbortController()
-    const timer = setTimeout(() => controller.abort(), this.#config.timeoutMs)
+    const timer = setTimeout(() => controller.abort(), timeoutMs)
 
     let envelope: { status?: string; response?: unknown; errorMessage?: string }
     try {
       const res = await this.#fetch(`${server.url}/api/${path}?${query.toString()}`, {
+        ...init,
         signal: controller.signal,
         headers: { accept: 'application/json' },
       })
       envelope = (await res.json()) as typeof envelope
     } catch (e) {
-      throw new UpstreamError('unreachable', this.#describe(e, [server.token, token]))
+      throw new UpstreamError('unreachable', this.#describe(e, [server.token, token], timeoutMs))
     } finally {
       clearTimeout(timer)
     }
@@ -72,11 +105,6 @@ export class Upstream {
           : (envelope?.errorMessage ?? `status ${String(envelope?.status)}`)
       throw new UpstreamError('rejected', why)
     }
-
-    if (this.#config.cacheSeconds > 0) {
-      if (this.#cache.size >= MAX_CACHED) this.#prune()
-      this.#cache.set(key, { until: this.#now() + this.#config.cacheSeconds * 1000, response: envelope.response })
-    }
     return envelope.response
   }
 
@@ -88,8 +116,8 @@ export class Upstream {
   }
 
   /* A short reason without anything that could carry the token. */
-  #describe(e: unknown, secrets: string[]): string {
-    if (e instanceof Error && e.name === 'AbortError') return `timed out after ${this.#config.timeoutMs} ms`
+  #describe(e: unknown, secrets: string[], timeoutMs: number): string {
+    if (e instanceof Error && e.name === 'AbortError') return `timed out after ${timeoutMs} ms`
     const cause = e instanceof Error ? (e.cause as { code?: unknown } | undefined) : undefined
     const text = typeof cause?.code === 'string' ? cause.code : e instanceof Error ? e.message : 'request failed'
     return secrets.reduce((out, secret) => (secret === '' ? out : out.split(secret).join('***')), text)