EvoFirewall/docs/runtime-validation.md

Runtime validation checkpoint

Docker/Linux runtime and RouterOS hardware are unavailable in this Windows workspace. No image smoke, load result, memory reduction or performance improvement is claimed. Docker Hub digest resolution was attempted on 2026-10-24 but DNS lookup failed. A checked-in base-image digest is still an unresolved release gate.

Local verification: node --test scripts/ci/sqlite-backup.test.mjs passed on Windows / Node 25.5.0 (1 test): committed WAL contents, refusal to overwrite a backup, offline restore requirement, removal of stale sidecars and preservation of the old destination when the input is invalid. Windows sandbox denied atomic rename; the identical test passed after an approved sandbox escalation. Bash syntax and workflow YAML parsing passed. These checks do not verify Alpine, Node 22 or the container image.

Containerized Gitea Linux-agent runner correction

The first pushed linux-agent-smoke job failed at unshare: Operation not permitted: its Gitea job container did not have the kernel capabilities needed by the nested namespace harness. The workflow now builds scripts/ci/linux-agent-smoke.Dockerfile and launches it using linux-agent-container-smoke.sh. Prerequisites and the checked-out agent/fixture/harness are copied into the helper image, so a remote Docker daemon needs no runner filesystem bind mounts.

The helper has its own network/PID/mount namespaces, --network none, a read-only image filesystem and a bounded temporary filesystem. It drops all capabilities except SYS_ADMIN (namespace/mount operations), NET_ADMIN (veth, nftables and ipset inside the test namespaces), and NET_RAW (the iptables set extension opens a raw socket while querying ipset). It retains Docker's default seccomp policy, enables no-new-privileges and never receives privileged mode, host networking/PIDs, host mounts or a Docker socket. On AppArmor-enabled daemons only this helper uses apparmor=unconfined, because Docker's default AppArmor policy denies the required mounts. See Docker's capability guidance and AppArmor policy guidance.

The original harness still creates and verifies separate network/mount/PID namespaces before firewall mutations. A preflight namespace probe fails clearly when the daemon/kernel does not support the required isolation; there is no skip, privileged retry or host-firewall fallback. The daemon must support namespaces, IPv6, veth, nftables, iptables and ipset; the helper cannot load missing host kernel modules. Its immutable image ID is used for execution and its container/image are cleaned up while preserving the original test exit code.

Because the harness masks /etc, the helper resolves all six Debian iptables/ip6tables/save/restore alternatives to direct binary links at build time. Only public protocol/service name databases from the helper image are restored into the private /etc; no runner /etc files or credentials are copied. Local validation covers YAML and shell syntax, the fixture ACK unit test and a fake-Docker wrapper regression (isolation flags, build-context behavior, cleanup and nonzero exit propagation). Docker is unavailable locally, so the corrected native helper run remains pending CI.

Image build and publication

2026-10-26 checkpoint: Docker CLI is still absent and wsl --status reports WSL is not installed. Local checks passed for the full-size fresh fixture/reuse refusal, selected-platform manifest validation and generated-schema drift check. These are not native container results. The load runner requires zero telemetry history/watermarks/totals, checks three unique raw reports and 200/400 deltas per agent plus 20000/40000 fleet totals, and reports report-ingestion and timeseries p95 independently. The first v2 counter establishes a baseline, so it contributes no fabricated traffic.

Both run-scoped CI builders now load deploy/docker/buildkitd.toml: GC targets 12GB maximum cache usage with 2GB reserved and 5GB free-space target; transient local/cachemount/git data has a 48-hour/1GB policy. These are garbage-collection targets rather than hard filesystem quotas. Configuration fields follow the BuildKit daemon reference. The real runner must still prove support and effective cleanup. Existing mirror tags are revalidated by comparing their selected platform manifest digest with the immutable upstream descriptor before use; a mismatched or missing platform fails publication.

The deployment remains one Alpine container: Fastify, SPA and SQLite. The Dockerfile has independent api-build and web-build branches sharing the dependency stage. pnpm deploy --prod creates the API dependency closure; the API root uses an allowlist (dist, node_modules, package.json). DB package files preserves migrations and shared runtime exports. Agent scripts come only from the API's dist/agent-scripts build output. Build tools stay in the dependency stages. Native module loading and migrations are checked by image smoke, not inferred from a successful TypeScript build.

Bake exports one registry cache from the final target with mode=max, covering both reachable branches. Pull requests import the release cache and explicitly disable export; forks build without registry credentials (private cache reads may miss). Image smoke runs on relevant PR and main/master changes. CD publication is serialized using workflow concurrency; the same registry cache has one writer. Gitea runner support for concurrency must be confirmed on the first run.

The mirror script resolves Docker Hub's manifest digest, gives each source digest its own mirror tag, then resolves and uses the mirror by digest. Resolution/copy errors fail CD instead of silently selecting a stale mirror. BASE_NODE_SOURCE can supply an approved node:22-alpine@sha256:... once independently verified. Defaults in Dockerfile/bake remain a mutable tag until that reviewed digest is available. Record and review the BASE_NODE line from CI.

cd deploy/docker
CACHE_REF_NODE=git.shx.one/denozord/evofw-buildcache:node-buildcache \
  docker buildx bake -f docker-bake.hcl --set evofw.cache-to= \
  --set evofw.tags=evofw-smoke:local --load default
cd ../..
sh scripts/ci/image-smoke.sh evofw-smoke:local
REQUIRE_UPGRADE=true sh scripts/ci/image-smoke.sh evofw-smoke:local \
  git.shx.one/denozord/evofw@sha256:PREVIOUS_DIGEST

Smoke uses disposable volumes and verifies migration/native SQLite startup, /health, DB-backed /ready, API JSON, SPA deep links, agent script delivery, runtime file placement, clean SIGTERM, restart persistence, an online backup and restoration into another volume. Every CD quality invocation builds and runs smoke, including docs-only releases, before semantic-release can publish. The image healthcheck uses /ready with 40s startup grace; Compose allows 30s shutdown and rotates JSON logs (3 x 10MB).

Upgrade baseline selection is explicit and fail-closed. A configured Gitea variable EVOFW_PREVIOUS_IMAGE takes precedence and must be an immutable repository@sha256:... reference; use the actually deployed digest when that is the intended rehearsal baseline. Mutable overrides fail rather than silently selecting another image. When the variable is unset for a release run, scripts/ci/resolve-upgrade-baseline.sh pulls git.shx.one/<lowercase-owner>/evofw:latest, requires exactly one matching repository digest from Docker's RepoDigests, and passes that immutable reference to the unchanged smoke guard. This is the previously published image, which may differ from production. A pull failure, missing/unrelated/ambiguous digest or invalid reference fails CI; it never enables a first-release exception automatically. Registry access uses the existing anonymous/public pull path and does not expose registry secrets to PRs.

Only an explicitly approved first deployment may set EVOFW_FIRST_RELEASE=true; unset it immediately afterward. Without an explicit baseline, PR runs do not pull a published image and can perform development smoke with upgrade validation reported NOT RUN. A pinned override is honored for PRs and for explicitly configured first-release runs. Local shell syntax and the fake-Docker resolver regression passed, covering override priority, latest-to-digest resolution, pull failure, wrong/ambiguous repository digests, mutable override rejection, PR behavior and explicit first release. The real upgrade still requires CI execution; no new native pass is inferred from these checks.

SQLite backup and recovery

sh deploy/docker/backup.sh evofirewall /secure-backups/evofw-2026-10-24.db
docker compose stop evofirewall
sh deploy/docker/restore.sh git.shx.one/denozord/evofw:vVERSION \
  PROJECT_evofw-data /secure-backups/evofw-2026-10-24.db
docker compose up -d evofirewall

Backup uses better-sqlite3's SQLite online backup API and verifies the output, including foreign keys. It includes committed WAL data while the API is running; copying only app.db from a live WAL database is not a backup. The host output must be new. Keep backups on a separate durable filesystem and protect them like credentials.

Restore supports a named Docker volume with /data/app.db. Stop all writers and prevent supervisors/other operators from starting them for the entire restore. The wrapper first verifies the volume exists and has no running writers. It creates a stopped, networkless helper with the named volume, copies the local backup into its private /tmp through docker cp, then checks again for running writers immediately before starting the helper. Failed Docker writer enumeration is an error, never evidence that the volume is offline. The helper verifies the input, prepares a temporary database, removes old sidecars and replaces the offline DB. Start/wait failures and the restore process exit status propagate; cleanup removes only the helper and its anonymous volumes, never the target named volume. No runner-local path is used as a daemon-host bind mount, so a remote Docker daemon is supported. The offline checks are guards, not a cross-host lock. Take a backup before replacing an existing DB. Rehearse restoration and upgrade on a cloned volume before production. Downgrading application code after new migrations requires restoring the matching pre-upgrade backup; down migrations are not supplied.

The subsequent pushed image smoke confirmed upgrade persistence, API/SPA/script checks, online backup integrity and restart-marker persistence. Restore then failed because a runner-local temporary backup path was passed as a bind source to a different Docker daemon host. The stopped-helper/copy correction addresses that transport failure without weakening SQLite verification. A fake-Docker regression passed for successful restore, copy/start/process failures, initially active and newly active volumes, failed writer enumeration at both guards, cleanup and target-volume preservation; CI now runs it beside the backup/marker tests. Actual restore completion and the downstream constrained benchmark still require the corrected CI run. Image-smoke cleanup also checks container existence before collecting final logs, avoiding misleading missing-container errors after a successful stop/remove.

Reproducible baseline

Use the same Linux host, pinned image digest, Docker version and fixture for each candidate. The harness limits the disposable container to the target 2 vCPU / 4GB and records host/Docker/image metadata. It does not add production CPU/heap caps based on guesses.

sh scripts/bench/runtime.sh evofw-smoke:local runtime-artifacts/before
node scripts/bench/bundle.mjs apps/web/dist > runtime-artifacts/bundle.json

fixture.mjs seeds 100 approved agents assigned to one 50,000-prefix policy on a fresh migrated DB. The load runner fetches each agent's policy and heartbeat at configurable concurrency (default 10), then imports 100,000 prefixes through HTTP into a second list. It records statuses, payload bytes, per-request duration, p50/p95/max, raw Docker CPU/memory/block IO samples, process RSS/IO, DB/WAL sizes, cold readiness seconds and a verified backup. Every HTTP error fails the harness; a payload limit failure is evidence to resolve, not a passed capacity test. Collection runs once per disposable fixture; retain JSON/logs and backup according to the test environment's data policy.

The bundle tool records every JS/CSS output file's raw and gzip size; totals include lazy chunks and are not the initial route transfer size. Browser navigation/network traces remain required for route-level performance. Repeat at least three times per candidate and compare median runs, plus cold/warm policy behavior, while holding environment constant. Current measurement table: startup, RSS, CPU, IO, WAL, 100-agent policy latency and 100k import are pending Docker execution; bundle output may be collected locally after a successful web build. No gain percentage is available yet.

The migrated fixture itself passed a local test at full 100-agent / 50k-prefix scale, including its valid priority constraint, foreign keys and all 100k unique import CIDRs. This does not execute the image or HTTP capacity profile. The load runner separates 1000 cold/burst requests from a paced 400-request steady profile after warmup; verification and import add separate requests. Every HTTP error including 429 fails the runner. A separate in-process 800-request regression passed; the full Linux resource run is still pending. See operational-hardening.md for the isolated namespace CI and guard semantics. Root separately verified the ReUI premium CLI @reui/settings-2 --dry-run successfully (10 hypothetical files, no installation changes), with the skill at 68f0e6fa91.

Steady defaults are two workers, one agent cycle per worker every 1200ms (approximately 100 agents/minute), with a new sequence-4 report, its exact retry, a summary read and a timeseries read per agent. It measures accepted ingestion separately from duplicate retries. Both phases assert response ACK identity and duplicate flags. Post-burst verification requires three raw rows and 200/400 dropped/accepted per agent; post-steady verification requires four raw rows and 300/600 per agent, fleet totals 30000/60000. Fresh complete runs now issue 1611 requests including warmup, verification, import and diagnostics.

BENCH_STEADY_CONCURRENCY and BENCH_STEADY_INTERVAL_MS expose the steady profile. BENCH_P95_LIMIT_MS is optional for direct local runs; when set it fails on steady new ingestion or timeseries p95 above the limit. The constrained 2-vCPU/4GB Docker harness sets a default 500ms limit. Cold/burst p95 is reported independently and does not contaminate that acceptance check. Shorter pacing or extra requests from the same user can deliberately trigger UI limits; any 429 still fails the run. The target p95 acceptance remains pending the actual constrained CI run.

The quality workflow now actually invokes scripts/bench/runtime.sh after built-image smoke. Bake loads a run-unique image tag; CI captures its immutable local image ID and passes that exact ID to both smoke and the benchmark. The host load driver uses Node 22; the harness creates its own disposable 2-vCPU/4GB volume/container. The step explicitly sets the 500ms limit, concurrency 10 for burst and two workers/1200ms pacing for steady, with a 15-minute timeout. There are no production credentials in the fixture or request measurements. Relevant PRs and every CD invocation run the Docker job, so a failed constrained benchmark blocks release. This is CI wiring evidence; no native result is claimed before that workflow executes.

Remaining gates

• Build and run the image on Linux/amd64, including the native binding, production-only dependency closure and the actual Gitea runner.
• Pin a reviewed and resolved Alpine Node digest in source; verify mirror provenance and registry cache behavior.
• Run previous-production-image upgrade and backup recovery with a representative database; preserve the immutable pre-upgrade backup.
• Run the fixture baseline and inspect failed HTTP responses, event-loop behavior and DB growth. Long-duration retention/capacity testing and Linux/MikroTik packet/application validation remain separate.

Local functional load observation (2026-10-26)

Root executed the benchmark HTTP load against a Windows Node 25.5 in-memory DB process, without CPU/RAM quotas. At concurrency 10, the fixture had 100 same-NAT agents and a 50k-prefix policy. All 1107 measured requests succeeded with zero HTTP failures and zero 429s. Each agent submitted three unique v2 reports, each retried once (six report POSTs), and retained exactly three raw rows. Assertions passed for per-agent 200 dropped/400 accepted deltas, fleet totals 20000/40000 and a 100k-prefix import.

Observed p95: report ingestion 30.08ms, timeseries 371.10ms, summary 16.94ms, initial policy reads 922.63ms and heartbeat 784.70ms. After the run at 122s uptime, the process reported RSS 177385472 bytes and heap-used 38631240 bytes. These are a final snapshot, not peak memory. This Windows functional run is neither a Linux/Alpine image result nor a 2-vCPU/4GB target-host baseline; no performance improvement or comparison is established.

The initial import response was 21903744 bytes and took 2.74s, exposing response amplification. Backend follow-up changes it to a bounded summary; this observation predates that change and does not validate the corrected response size. Preserve a fresh-run result for the revised endpoint before comparing payload or timing. Native CI still must execute image startup, SQLite binding/backup/upgrade, resource sampling and both isolated firewall backends.

A second fresh Windows in-memory run of the original mixed profile passed all 1107 requests with zero 429s and all correctness checks, including the corrected 120-byte import summary. Its p95 was reports 60.54ms, timeseries 647.73ms, cold policy 2221ms and summary 38.17ms. Concurrent development tools were active and CPU/RAM were unbounded. The timeseries result exceeds 500ms; it is not evidence that the target passes. These two runs mixed cold policy bursts and reads; neither measures the newly separated paced steady profile or establishes a before/after performance comparison.

The final fresh run used the separated profiles on Windows Node 25.5 with an in-memory DB and no CPU/RAM quota. All 1611 requests passed with zero rate limits, exact report-ID/sequence and duplicate ACK checks, four raw rows per agent, final per-agent 300 dropped/600 accepted and fleet totals 30000/60000. The 100k import summary remained below 256 bytes. Burst profile: 1000 requests at concurrency 10, new-ingestion p95 372.753ms (300 samples), timeseries p95 782.623ms and cold-policy p95 3451.412ms. Steady profile: 400 requests, two workers paced at 1200ms; new ingestion 100 samples p95 11.0855ms/max 14.4917ms, timeseries 100 samples p95 8.0675ms/max 10.7258ms, duplicate retries 100 samples p95 7.5481ms and summaries 100 samples p95 3.6905ms. This verifies the revised harness and local correctness. It does not prove native 2-vCPU/4GB capacity or establish a performance improvement. The actual constrained CI threshold remains pending execution.

Builder concurrency correction: each quality/publication job uses its own run-ID/attempt builder, with setup-buildx cleanup enabled. The registry cache is durable; local builder state is disposable. The verification script inspects the actual container configuration before use. Jobs never remove a shared builder, so concurrent PRs cannot tear down another build. The old persistent evofw builder is no longer selected; remove its stale container/state only during an idle-runner maintenance window after confirming no legacy job is using it.

The subsequent ipset smoke failed with iptables-restore ... Can't open socket to ipset. The iptables set extension source opens an AF_INET raw socket in get_version before querying the set API. The Linux raw socket manual requires CAP_NET_RAW for this operation. The helper therefore adds only NET_RAW to its existing capabilities; it keeps its isolated network, default seccomp, and all other restrictions. The wrapper regression checks the exact three-capability set, exit propagation, and cleanup. This source-based fix still requires the next Linux CI smoke to confirm native ipset execution; the Windows test does not execute Docker or kernel commands.

The published upgrade baseline subsequently started successfully and returned /ready 200, but the smoke marker failed while resolving better-sqlite3 through a synthetic /app/node_modules/@evofw/db/package.json path. The marker now creates one uniquely named static list with one CIDR through the public API, captures its ID and verifies that same ID/name/entry across upgrade, restart and restoration. Later verification never recreates a missing marker. This API shape was checked against the previous aa47584 list routes and accepts their HTTP 200 creation response as well as newer HTTP 201 responses. It makes no assumptions about the old image's dependency layout; native SQLite startup and persistence are still exercised by readiness and actual API writes/reads.

The backup utility and benchmark fixture had the same pnpm resolution assumption. Both now resolve the DB package's real filesystem location before constructing its dependency resolver. The backup regression uses an isolated pnpm-shaped symlink/junction tree, removes the former NODE_PATH workaround, demonstrates that the synthetic resolver fails and verifies the real fixture loader plus WAL backup/offline restore. That regression passed locally after the required Windows sandbox escalation. The marker regression runs the actual embedded Node program with a fake public API: old/new response codes, retained ID, missing rows and lost entries are covered and passed. CI runs this marker regression beside the backup test, and the benchmark wrapper regression runs before its real image benchmark. These local checks do not establish a successful native upgrade/backup/benchmark; those remain pending CI. The nft namespace backend has already passed native IPv4/IPv6 denial, drift repair, invalid-candidate preservation and unrelated-table preservation; ipset and the full built-image sequence still require the corrected run.