EvoFirewall/docs/backend-hardening.md

# Backend hardening checkpoint

Completed in the first implementation batch:

- Shared browser-compatible strict IP/CIDR parser: mask network bits, canonical IPv6 spelling, IPv4-mapped IPv6 support; ingress validation for rules, overrides, port ACL, manual entries and reported IP/network hits.
- Manual list creation materializes and validates config entries through the same path as the entry editor.
- EvoBGP autocomplete decrypts the stored API token; community resolution and prefix refresh collect all cursor pages and reject broken/repeated cursors.
- Invalid JSON feed shapes, malformed prefixes and partial fetches retain the complete last successful entries. DNS/manual refresh failure retains the previous materialization.
- Canonical list snapshots skip identical replacements; unchanged refresh skips agent generation updates. Effective policy hashes exclude generation. Hostname rules skip unchanged cache replacement. The full affected list closure is staged before one transaction publishes configs, materializations and generation changes; parent DNS failure cannot leave a partially updated child.
- Replacement inserts use 200-entry SQL batches; regression covers 50,000 prefixes.
- Referenced list deletion returns conflict while nested lists, policy rules or port ACL depend on it.
- Self-cloning and cross-agent override deletion are rejected. Reordering validates an exact permutation and uses an available priority slot to avoid collisions with existing high priorities. Clone writes are transactional.
- Each SQL migration and its applied stamp commit atomically. Foreign-key enforcement is disabled outside the transaction for referenced-table rebuilds, checked before commit, and restored after success/failure. Populated upgrade regression verifies preserved children.

Validation at this checkpoint:

- Shared build and 22 tests pass; DB build passes; API TypeScript check passes.
- API focused backend suite: 29 tests pass, including eight new integrity regressions (encrypted autocomplete and manual config ingress included).
- The broader API run also exercised concurrent agent work. Remaining failures at that time were native Bash startup in the sandbox (four shell tests) and the old MikroTik renderer fixture (owned by the agent implementation wave). The legacy IP-hit network compatibility failure found in that run was fixed and the IP statistics suite now passes.

Second checkpoint: telemetry and diagnostics

- `applyReportV2BodySchema` / `ApplyReportV2Body` are exported from shared. `/v1/agent/apply-report` validates the complete v2 envelope. Invalid v2 cannot silently fall back to legacy handling. Negative, fractional and unsafe integer counters are rejected; canonical duplicate hit keys are rejected.
- Required v2 metadata: report ID, persistent sequence, boot ID, observation time, interval, independent traffic/IP/port epochs, capabilities and quality. Optional detail found/sent counts are validated against supplied arrays and persisted. Missing counts mark coverage partial; fewer sent elements mark truncation.
- One synchronous transaction performs replay/order checks, baseline updates, traffic/detail accumulation, samples, capabilities/quality, apply diagnostics, and host snapshot updates. HTTP 200 v2 acknowledgements echo `report_id` and `sequence`; `accepted: false` reports duplicate or stale delivery without repeating accounting.
- Durable agent replay/baseline state survives UI resets and raw-sample deletion. Once v2 is established, delayed legacy payloads cannot replace its state. Legacy-only telemetry remains accepted with explicit limited quality metadata.
- Boot/domain epoch changes establish new baselines even when absolute counters increase. Missing values and initial/new-epoch traffic baselines store nullable deltas rather than observed zeroes. Backward observation clocks mark the report partial and leave interval coverage unknown. Unsafe accumulated totals retain their exact previous value and mark overflow unsupported/partial instead of rounding.
- UI reset clears displayed traffic and detail totals while retaining device baselines, sequence watermark and history. Regression covers `100 -> reset -> 100 -> 105`, producing zero additional packets until the last report adds five.
- Agent DTO adds `diagnostics` with desired/applied revision/hash, version, capabilities, contact/telemetry/apply times, structured errors and convergence state. The desired hash is known after policy delivery for the current generation; stale generations expose a null desired hash. Error-only reports preserve the last successful applied acknowledgement. Counter-only telemetry cannot clear a previous apply failure.
- `/api/v1/ui/summary` returns the agreed compact counts. It authenticates `fw` access and independently omits each unauthorized dashboard/agents/lists/policies section. Shared `UiSummary` / `uiSummarySchema` are exported.
- API build no longer copies agent scripts twice through tsup `--publicDir`; explicit `dist/agent-scripts` packaging remains.

Second-checkpoint validation:

- Shared and DB builds pass; shared tests: 22/22; API TypeScript check passes.
- Dedicated telemetry suite: 10/10, summary/authentication/reset routes: 2/2.
- Existing backend suite plus new tests: 66/68 in the concurrent source-hardening checkout. The two failures were source transport mocks still targeting global fetch after the source owner introduced `safeFetchJson`; source owner is adapting those fixtures. Shell/device tests remain owned by the agent implementation wave.

Third checkpoint: analytics, retention and import bounds

- Fixed both second-review findings: an acknowledgement with the same hash/revision recovers after `status: error` even when no optional error text was supplied; unknown/reset/decreased IP/port counter intervals mark quality partial. Both regressions pass through the shared ingress schema.
- Migration 013 adds per-metric baseline timestamps, raw presence/interval metadata, an idempotent analytics-processed marker, and indexed per-agent UTC minute/hour/day buckets.
- Every accepted report publishes its sample, replay state and additive rollups in the same transaction. Counters span their own last accepted observation; missing values preserve that timestamp. Skipped sequence deltas use the whole observation span rather than the producer's shorter current sampling interval.
- Interval deltas are allocated proportionally across overlapping UTC buckets. Distribution is explicitly estimated when the observation interval differs from the bucket. Missing selected-agent metrics remain null; measured zero remains zero. Coverage is conservative across accepted/dropped metrics. Unknown first/new-epoch intervals are gaps. Future/backward clocks are partial and cannot create unbounded bucket loops.
- `GET /api/v1/stats/timeseries` and `.csv` accept `from`, `to`, `resolution=1m|1h|1d`, and comma-separated `agent_ids` (maximum 100). Ranges are half-open `[from,to)` and returned bounds are rounded to UTC bucket boundaries. The 1440-point cap applies after rounding. SQL aggregates the bounded persisted buckets; requests do not scan raw history. Shared `TimeseriesResponse` / `TimeseriesPoint` / `StatsResolution` are exported.
- `GET /api/v1/agents/:id/stats/raw` uses a stable `(recorded_at,id)` cursor, limit 1..500, `has_more` and `next_cursor`. It exposes nullable missing counters, deltas, interval starts, observation metadata, quality and detail counts. All analytics/CSV/raw reads require statistics permission.
- Retention runs hourly and at startup: raw/minute 24 hours, hourly 30 days, daily 90 days, detail 7 days. Existing raw samples are backfilled once as explicitly estimated/partial before deletion; atomic processed claims prevent overlapping workers from double accounting. New samples are already rolled up transactionally. The worker yields between bounded deletion/backfill batches and accepts shutdown cancellation. The old `STATS_RETENTION_DAYS` config field remains for compatibility but does not override these fixed tier policies.
- IP/port aggregates retain SQL top-N behavior and enforce per-agent storage budgets (2000 IP rows, 5000 port rows). Eviction marks truncated/partial quality. UI counter reset preserves rollups/history and device baselines.
- Only list creation/entry import routes accept 8 MiB bodies; both plaintext token and structured item imports are capped at 100000 entries. Other routes retain their previous body limit.

Third-checkpoint validation:

- Shared/DB builds and API TypeScript check pass. Focused backend run: 19 files, 88 tests pass, excluding native agent/sandbox socket suites.
- Dedicated analytics: 10 tests, telemetry: 12 tests, summary/routes/import: 4 tests. Coverage includes proportional multi-agent intervals, missing-metric spans, skipped sequences, epoch gaps, known zero, transactional rollback, huge intervals, retention tiers, estimated backfill, concurrent backfill, cardinality, CSV/RBAC/raw cursors, and an actual >1 MiB import of 100000 prefixes.
- Broader sandbox execution also reached new native agent/socket tests; the remaining failures were Bash startup, Windows atomic-file replacement restrictions and sandbox socket errors. The full unrestricted integration run belongs to the final root verification checkpoint.

Still pending; do not treat the roadmap as completed:

- Final frontend analytics verification and updated OpenAPI/reference documentation for the new telemetry and statistics contracts.
- Larger policy compilation/performance wave, source scheduling/retry observability and full target-load verification.
- Real Linux kernel/firewall and RouterOS acceptance; mock/static checks cannot establish production device behavior.

The backend schema, persistence, analytics and retention above are implemented. Cache/load acceptance and production kernel/device validation remain separate work.

F1 checkpoint: explain and temporary overrides are implemented and described in `docs/policy-explain.md` (migration015). Analytics follow-up regressions cover explicit v2-null backfill, interval loss counts, traffic-only completeness, preserved quality reasons and exact minute-to-hour aggregation. The later ipset explanation correction follows its actual INPUT-only compiler rather than assuming nft stateful/forward behavior.

Policy bounds/cache checkpoint: implemented50000 effective/200000 raw memberships,1000 expanded Port ACL rules,8 MiB policy responses, last-good-preserving refusal, dualstack `all` sources, DB-scoped128-entry/32 MiB conservative LRU/intern accounting, expiry/dependency invalidation and full-response versioned ETags. Opt-in RouterOS budget envelopes and durable `sequence_floor` ACK recovery are documented in `docs/policy-budget-cache.md`. Focused verification:88 distinct backend tests pass, API TypeScript/shared/DB builds pass. Conditional304 client bundles remain the agent owner's follow-up; existing clients without a conditional header receive200.

Still pending: prepared/batched IP/port detail UPSERT optimization and target-load profiling; snapshots/canary workflows; enrollment rotation and labels/groups; final OpenAPI synchronization and real kernel/RouterOS runtime acceptance. No broader roadmap completion is claimed.