EvoFirewall/docs/runtime-validation.md

Runtime validation checkpoint

Docker/Linux runtime and RouterOS hardware are unavailable in this Windows workspace. No image smoke, load result, memory reduction or performance improvement is claimed. Docker Hub digest resolution was attempted on 2026-10-24 but DNS lookup failed. A checked-in base-image digest is still an unresolved release gate.

Local verification: node --test scripts/ci/sqlite-backup.test.mjs passed on Windows / Node 25.5.0 (1 test): committed WAL contents, refusal to overwrite a backup, offline restore requirement, removal of stale sidecars and preservation of the old destination when the input is invalid. Windows sandbox denied atomic rename; the identical test passed after an approved sandbox escalation. Bash syntax and workflow YAML parsing passed. These checks do not verify Alpine, Node 22 or the container image.

Containerized Gitea Linux-agent runner correction

The first pushed linux-agent-smoke job failed at unshare: Operation not permitted: its Gitea job container did not have the kernel capabilities needed by the nested namespace harness. The workflow now builds scripts/ci/linux-agent-smoke.Dockerfile and launches it using linux-agent-container-smoke.sh. Prerequisites and the checked-out agent/fixture/harness are copied into the helper image, so a remote Docker daemon needs no runner filesystem bind mounts.

The helper has its own network/PID/mount namespaces, --network none, a read-only image filesystem and a bounded temporary filesystem. It drops all capabilities except SYS_ADMIN (namespace/mount operations) and NET_ADMIN (veth, nftables and ipset inside the test namespaces). It retains Docker's default seccomp policy, enables no-new-privileges and never receives privileged mode, host networking/PIDs, host mounts or a Docker socket. On AppArmor-enabled daemons only this helper uses apparmor=unconfined, because Docker's default AppArmor policy denies the required mounts. See Docker's capability guidance and AppArmor policy guidance.

The original harness still creates and verifies separate network/mount/PID namespaces before firewall mutations. A preflight namespace probe fails clearly when the daemon/kernel does not support the required isolation; there is no skip, privileged retry or host-firewall fallback. The daemon must support namespaces, IPv6, veth, nftables, iptables and ipset; the helper cannot load missing host kernel modules. Its immutable image ID is used for execution and its container/image are cleaned up while preserving the original test exit code.

Because the harness masks /etc, the helper resolves all six Debian iptables/ip6tables/save/restore alternatives to direct binary links at build time. Only public protocol/service name databases from the helper image are restored into the private /etc; no runner /etc files or credentials are copied. Local validation covers YAML and shell syntax, the fixture ACK unit test and a fake-Docker wrapper regression (isolation flags, build-context behavior, cleanup and nonzero exit propagation). Docker is unavailable locally, so the corrected native helper run remains pending CI.

Image build and publication

2026-10-26 checkpoint: Docker CLI is still absent and wsl --status reports WSL is not installed. Local checks passed for the full-size fresh fixture/reuse refusal, selected-platform manifest validation and generated-schema drift check. These are not native container results. The load runner requires zero telemetry history/watermarks/totals, checks three unique raw reports and 200/400 deltas per agent plus 20000/40000 fleet totals, and reports report-ingestion and timeseries p95 independently. The first v2 counter establishes a baseline, so it contributes no fabricated traffic.

Both run-scoped CI builders now load deploy/docker/buildkitd.toml: GC targets 12GB maximum cache usage with 2GB reserved and 5GB free-space target; transient local/cachemount/git data has a 48-hour/1GB policy. These are garbage-collection targets rather than hard filesystem quotas. Configuration fields follow the BuildKit daemon reference. The real runner must still prove support and effective cleanup. Existing mirror tags are revalidated by comparing their selected platform manifest digest with the immutable upstream descriptor before use; a mismatched or missing platform fails publication.

The deployment remains one Alpine container: Fastify, SPA and SQLite. The Dockerfile has independent api-build and web-build branches sharing the dependency stage. pnpm deploy --prod creates the API dependency closure; the API root uses an allowlist (dist, node_modules, package.json). DB package files preserves migrations and shared runtime exports. Agent scripts come only from the API's dist/agent-scripts build output. Build tools stay in the dependency stages. Native module loading and migrations are checked by image smoke, not inferred from a successful TypeScript build.

Bake exports one registry cache from the final target with mode=max, covering both reachable branches. Pull requests import the release cache and explicitly disable export; forks build without registry credentials (private cache reads may miss). Image smoke runs on relevant PR and main/master changes. CD publication is serialized using workflow concurrency; the same registry cache has one writer. Gitea runner support for concurrency must be confirmed on the first run.

The mirror script resolves Docker Hub's manifest digest, gives each source digest its own mirror tag, then resolves and uses the mirror by digest. Resolution/copy errors fail CD instead of silently selecting a stale mirror. BASE_NODE_SOURCE can supply an approved node:22-alpine@sha256:... once independently verified. Defaults in Dockerfile/bake remain a mutable tag until that reviewed digest is available. Record and review the BASE_NODE line from CI.

cd deploy/docker
CACHE_REF_NODE=git.shx.one/denozord/evofw-buildcache:node-buildcache \
  docker buildx bake -f docker-bake.hcl --set evofw.cache-to= \
  --set evofw.tags=evofw-smoke:local --load default
cd ../..
sh scripts/ci/image-smoke.sh evofw-smoke:local
REQUIRE_UPGRADE=true sh scripts/ci/image-smoke.sh evofw-smoke:local \
  git.shx.one/denozord/evofw@sha256:PREVIOUS_DIGEST

Smoke uses disposable volumes and verifies migration/native SQLite startup, /health, DB-backed /ready, API JSON, SPA deep links, agent script delivery, runtime file placement, clean SIGTERM, restart persistence, an online backup and restoration into another volume. Supply Gitea variable EVOFW_PREVIOUS_IMAGE with the last deployed version/digest to exercise upgrade migrations. Every CD quality invocation now builds and runs smoke, including docs-only releases, before semantic-release can publish. EVOFW_PREVIOUS_IMAGE must end in an immutable @sha256:... digest. Missing or mutable previous images fail release validation. Only an explicitly approved first deployment may set Gitea variable EVOFW_FIRST_RELEASE=true; unset it immediately after that first deployment. PR/development smoke can run without upgrade validation and reports NOT RUN. The image healthcheck uses /ready with 40s startup grace; Compose allows 30s shutdown and rotates JSON logs (3 x 10MB).

SQLite backup and recovery

sh deploy/docker/backup.sh evofirewall /secure-backups/evofw-2026-10-24.db
docker compose stop evofirewall
sh deploy/docker/restore.sh git.shx.one/denozord/evofw:vVERSION \
  PROJECT_evofw-data /secure-backups/evofw-2026-10-24.db
docker compose up -d evofirewall

Backup uses better-sqlite3's SQLite online backup API and verifies the output, including foreign keys. It includes committed WAL data while the API is running; copying only app.db from a live WAL database is not a backup. The host output must be new. Keep backups on a separate durable filesystem and protect them like credentials.

Restore supports a named Docker volume with /data/app.db. Stop all writers and prevent supervisors/other operators from starting them for the entire restore. The wrapper rejects a volume currently mounted by any running container, mounts the backup read-only, verifies it, prepares a temporary database, removes old sidecars and replaces the offline DB. This exclusion check is a guard, not a cross-host lock. Take a backup before replacing an existing DB. Rehearse restoration and upgrade on a cloned volume before production. Downgrading application code after new migrations requires restoring the matching pre-upgrade backup; down migrations are not supplied.

Reproducible baseline

Use the same Linux host, pinned image digest, Docker version and fixture for each candidate. The harness limits the disposable container to the target 2 vCPU / 4GB and records host/Docker/image metadata. It does not add production CPU/heap caps based on guesses.

sh scripts/bench/runtime.sh evofw-smoke:local runtime-artifacts/before
node scripts/bench/bundle.mjs apps/web/dist > runtime-artifacts/bundle.json

fixture.mjs seeds 100 approved agents assigned to one 50,000-prefix policy on a fresh migrated DB. The load runner fetches each agent's policy and heartbeat at configurable concurrency (default 10), then imports 100,000 prefixes through HTTP into a second list. It records statuses, payload bytes, per-request duration, p50/p95/max, raw Docker CPU/memory/block IO samples, process RSS/IO, DB/WAL sizes, cold readiness seconds and a verified backup. Every HTTP error fails the harness; a payload limit failure is evidence to resolve, not a passed capacity test. Collection runs once per disposable fixture; retain JSON/logs and backup according to the test environment's data policy.

The bundle tool records every JS/CSS output file's raw and gzip size; totals include lazy chunks and are not the initial route transfer size. Browser navigation/network traces remain required for route-level performance. Repeat at least three times per candidate and compare median runs, plus cold/warm policy behavior, while holding environment constant. Current measurement table: startup, RSS, CPU, IO, WAL, 100-agent policy latency and 100k import are pending Docker execution; bundle output may be collected locally after a successful web build. No gain percentage is available yet.

The migrated fixture itself passed a local test at full 100-agent / 50k-prefix scale, including its valid priority constraint, foreign keys and all 100k unique import CIDRs. This does not execute the image or HTTP capacity profile. The load runner separates 1000 cold/burst requests from a paced 400-request steady profile after warmup; verification and import add separate requests. Every HTTP error including 429 fails the runner. A separate in-process 800-request regression passed; the full Linux resource run is still pending. See operational-hardening.md for the isolated namespace CI and guard semantics. Root separately verified the ReUI premium CLI @reui/settings-2 --dry-run successfully (10 hypothetical files, no installation changes), with the skill at 68f0e6fa91.

Steady defaults are two workers, one agent cycle per worker every 1200ms (approximately 100 agents/minute), with a new sequence-4 report, its exact retry, a summary read and a timeseries read per agent. It measures accepted ingestion separately from duplicate retries. Both phases assert response ACK identity and duplicate flags. Post-burst verification requires three raw rows and 200/400 dropped/accepted per agent; post-steady verification requires four raw rows and 300/600 per agent, fleet totals 30000/60000. Fresh complete runs now issue 1611 requests including warmup, verification, import and diagnostics.

BENCH_STEADY_CONCURRENCY and BENCH_STEADY_INTERVAL_MS expose the steady profile. BENCH_P95_LIMIT_MS is optional for direct local runs; when set it fails on steady new ingestion or timeseries p95 above the limit. The constrained 2-vCPU/4GB Docker harness sets a default 500ms limit. Cold/burst p95 is reported independently and does not contaminate that acceptance check. Shorter pacing or extra requests from the same user can deliberately trigger UI limits; any 429 still fails the run. The target p95 acceptance remains pending the actual constrained CI run.

The quality workflow now actually invokes scripts/bench/runtime.sh after built-image smoke. Bake loads a run-unique image tag; CI captures its immutable local image ID and passes that exact ID to both smoke and the benchmark. The host load driver uses Node 22; the harness creates its own disposable 2-vCPU/4GB volume/container. The step explicitly sets the 500ms limit, concurrency 10 for burst and two workers/1200ms pacing for steady, with a 15-minute timeout. There are no production credentials in the fixture or request measurements. Relevant PRs and every CD invocation run the Docker job, so a failed constrained benchmark blocks release. This is CI wiring evidence; no native result is claimed before that workflow executes.

Remaining gates

• Build and run the image on Linux/amd64, including the native binding, production-only dependency closure and the actual Gitea runner.
• Pin a reviewed and resolved Alpine Node digest in source; verify mirror provenance and registry cache behavior.
• Run previous-production-image upgrade and backup recovery with a representative database; preserve the immutable pre-upgrade backup.
• Run the fixture baseline and inspect failed HTTP responses, event-loop behavior and DB growth. Long-duration retention/capacity testing and Linux/MikroTik packet/application validation remain separate.

Local functional load observation (2026-10-26)

Root executed the benchmark HTTP load against a Windows Node 25.5 in-memory DB process, without CPU/RAM quotas. At concurrency 10, the fixture had 100 same-NAT agents and a 50k-prefix policy. All 1107 measured requests succeeded with zero HTTP failures and zero 429s. Each agent submitted three unique v2 reports, each retried once (six report POSTs), and retained exactly three raw rows. Assertions passed for per-agent 200 dropped/400 accepted deltas, fleet totals 20000/40000 and a 100k-prefix import.

Observed p95: report ingestion 30.08ms, timeseries 371.10ms, summary 16.94ms, initial policy reads 922.63ms and heartbeat 784.70ms. After the run at 122s uptime, the process reported RSS 177385472 bytes and heap-used 38631240 bytes. These are a final snapshot, not peak memory. This Windows functional run is neither a Linux/Alpine image result nor a 2-vCPU/4GB target-host baseline; no performance improvement or comparison is established.

The initial import response was 21903744 bytes and took 2.74s, exposing response amplification. Backend follow-up changes it to a bounded summary; this observation predates that change and does not validate the corrected response size. Preserve a fresh-run result for the revised endpoint before comparing payload or timing. Native CI still must execute image startup, SQLite binding/backup/upgrade, resource sampling and both isolated firewall backends.

A second fresh Windows in-memory run of the original mixed profile passed all 1107 requests with zero 429s and all correctness checks, including the corrected 120-byte import summary. Its p95 was reports 60.54ms, timeseries 647.73ms, cold policy 2221ms and summary 38.17ms. Concurrent development tools were active and CPU/RAM were unbounded. The timeseries result exceeds 500ms; it is not evidence that the target passes. These two runs mixed cold policy bursts and reads; neither measures the newly separated paced steady profile or establishes a before/after performance comparison.

The final fresh run used the separated profiles on Windows Node 25.5 with an in-memory DB and no CPU/RAM quota. All 1611 requests passed with zero rate limits, exact report-ID/sequence and duplicate ACK checks, four raw rows per agent, final per-agent 300 dropped/600 accepted and fleet totals 30000/60000. The 100k import summary remained below 256 bytes. Burst profile: 1000 requests at concurrency 10, new-ingestion p95 372.753ms (300 samples), timeseries p95 782.623ms and cold-policy p95 3451.412ms. Steady profile: 400 requests, two workers paced at 1200ms; new ingestion 100 samples p95 11.0855ms/max 14.4917ms, timeseries 100 samples p95 8.0675ms/max 10.7258ms, duplicate retries 100 samples p95 7.5481ms and summaries 100 samples p95 3.6905ms. This verifies the revised harness and local correctness. It does not prove native 2-vCPU/4GB capacity or establish a performance improvement. The actual constrained CI threshold remains pending execution.

Builder concurrency correction: each quality/publication job uses its own run-ID/attempt builder, with setup-buildx cleanup enabled. The registry cache is durable; local builder state is disposable. The verification script inspects the actual container configuration before use. Jobs never remove a shared builder, so concurrent PRs cannot tear down another build. The old persistent evofw builder is no longer selected; remove its stale container/state only during an idle-runner maintenance window after confirming no legacy job is using it.