test(netflow): зафиксировать приёмку на двух маршрутизаторах
This commit is contained in:
Denozordec committed
1 parent
0abdac82de
commit
fd2a42ddfe
3 files changed
+435
No files matched your search
@@ -154,3 +154,25 @@ docker run --rm --cap-drop ALL --cap-add NET_ADMIN --read-only --tmpfs /tmp \
|
||||
Она не заменяет приёмку на стенде с двумя MikroTik: handshake/IPFIX от каждого,
|
||||
сохранение первого соединения при добавлении второго, перезапуск агента/backend
|
||||
и холодная загрузка Linux-хоста с проверкой UDP bind.
|
||||
|
||||
### Приёмка на стенде — 2026-10-16
|
||||
|
||||
На `161.104.46.222` проверены временные контейнеры новой ветки с отдельной БД.
|
||||
15 тестов агента прошли с настоящим kernel WireGuard; 32 профильных скрипта
|
||||
и две отдельные SQL-проверки (monthly и IPv6 UDP/worker) завершились успешно.
|
||||
Оба MikroTik (`194.226.121.150`, `104.165.244.7`) подключены через API: подтверждены
|
||||
handshake, сохранение первого туннеля при добавлении второго и повторное подключение
|
||||
без дубликатов. В БД записаны IPFIX-потоки обоих MikroTik и синтетического Linux
|
||||
экспортёра через WireGuard. Реальные перезапуски контейнеров агента и backend
|
||||
сохранили соединения, приём пакетов продолжился.
|
||||
|
||||
При проверке исправлен API настроек: счётчики и время последнего пакета теперь
|
||||
читаются из работающего коллектора. После исправления полный живой сценарий повторён.
|
||||
Настройки обоих маршрутизаторов восстановлены; временные контейнеры, интерфейс,
|
||||
БД/роли и приватные файлы удалены. Основные сервисы остались на прежних образах.
|
||||
|
||||
Установщик с интеграцией `wg-quick`/systemd и холодная перезагрузка хоста на этом
|
||||
стенде не выполнялись. Образы собраны поверх существующего runtime backend;
|
||||
полная release-сборка frontend и публикация образов в эту проверку не входили.
|
||||
Результаты, исходные неудачные попытки и границы —
|
||||
[отчёт приёмки](ux-ui/acceptance/netflow-agent-2026-10-16.json).
|
||||
@@ -2758,3 +2758,14 @@ Runtime backend ux-regression222/frontend ux-routing218. Latest backend25/UI52
|
||||
2 roundtrip objects,12 unique API scenarios, synthetic native export/CHR contacts0. Own DB/roles/catalog/localfiles/currentobjects cleaned. Versioning Enabled:8versions+8markers removed,3 own prefixes remaining0; bucket configuration unchanged. Native backup/restore/live UI/смена storage origin/retention semantics остаются230. Private env удалены, secrets не в published files. [API](ux-ui/acceptance/s3-application229.json), [version cleanup](ux-ui/acceptance/s3-version-cleanup229.json).
|
||||
|
||||
Локальные .rsc после235: `/opt/mmapp-test/data/backups` → `/app/storage/backups`; см. [порядок переноса](../deploy/BACKUP-STORAGE.md). Две новые native copies после force-recreate проверены и удалены; десять старых catalog rows сохранены с зафиксированным отсутствием files.
|
||||
|
||||
## NetFlow host agent — 2026-10-16
|
||||
|
||||
Ветка `codex/netflow-host-agent`, исходники `db81cf0f` + исправление `0abdac82`.
|
||||
Временные backend/agent и отдельные БД: kernel WG 15/15, профиль32 и отдельные
|
||||
monthly/IPv6 UDP SQL passed. Оба CHR установили handshake и передали IPFIX в БД;
|
||||
повторное подключение, сохранение стороннего пира и рестарты agent/backend прошли.
|
||||
Найденные устаревшие счётчики API исправлены, живой сценарий повторён полностью.
|
||||
Исходные Traffic Flow настройки восстановлены, own WG/containers/DB/roles/keys
|
||||
удалены; четыре основных сервиса и их образы сохранены. Installer/systemd/cold reboot
|
||||
не выполнялись. [Отчёт и границы](ux-ui/acceptance/netflow-agent-2026-10-16.json).
|
||||
@@ -0,0 +1,402 @@
|
||||
{
|
||||
"date": "2026-10-16",
|
||||
"stand": "161.104.46.222",
|
||||
"branch": "codex/netflow-host-agent",
|
||||
"baseCommit": "db81cf0f",
|
||||
"testedSourceCommit": "0abdac82de3fd53fa0eeaae672a3681918191a9a",
|
||||
"passed": true,
|
||||
"scope": "Temporary backend/agent containers, separate PostgreSQL databases and host wg-flow; both authorized test MikroTik configured and restored.",
|
||||
"build": {
|
||||
"strategy": "Current compiled backend/contracts over existing backend runtime with matching application dependencies; test-only jose and transpiled tests added. Agent built from its Dockerfile.",
|
||||
"uploadedFilesVerified": 1976,
|
||||
"baseProductionCompiledFilesVerified": 1396,
|
||||
"patchedCompiledRouteSha256": "99a5b78f0586cbc53c35ed4bcefadd4fdd6ecea6da075f936b7768c634b03dd1",
|
||||
"patchedCompiledRouteMatchesLocal": true
|
||||
},
|
||||
"agentKernelTests": {
|
||||
"passed": 15,
|
||||
"failed": 0,
|
||||
"skipped": 0,
|
||||
"network": "Isolated container namespace",
|
||||
"coverage": [
|
||||
"Real kernel WireGuard apply",
|
||||
"Foreign persistent/runtime peers",
|
||||
"Unchanged interface identity",
|
||||
"Validation/conflicts",
|
||||
"Configuration and runtime rollback"
|
||||
]
|
||||
},
|
||||
"sql": {
|
||||
"primary": {
|
||||
"database": "mm_accept_166_1792139159368",
|
||||
"passed": true,
|
||||
"cases": [
|
||||
{
|
||||
"path": "services/traffic-flow-host-file-builders.test.js",
|
||||
"durationMs": 196,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-key-execution.test.js",
|
||||
"durationMs": 54,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "routes/traffic-flow-settings.test.js",
|
||||
"durationMs": 586,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "plugins/auth.netflow-settings.test.js",
|
||||
"durationMs": 665,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-agent-install.test.js",
|
||||
"durationMs": 174,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-agent-client.test.js",
|
||||
"durationMs": 260,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-operation-queue.test.js",
|
||||
"durationMs": 54,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "routes/traffic-flow-host.test.js",
|
||||
"durationMs": 830,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "routes/traffic-flow-query.test.js",
|
||||
"durationMs": 1816,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-month.test.js",
|
||||
"durationMs": 1146,
|
||||
"skips": [
|
||||
"Monthly SQL: skip (requires an owned isolated database)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-parse.test.js",
|
||||
"durationMs": 1174,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-ipv6-pipeline.test.js",
|
||||
"durationMs": 1203,
|
||||
"skips": [
|
||||
"IPv6 UDP/worker/SQL: skip (requires an owned isolated database)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-map-exporter.test.js",
|
||||
"durationMs": 55,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-ifaces.test.js",
|
||||
"durationMs": 64,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-ifindex.test.js",
|
||||
"durationMs": 59,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-dedup.test.js",
|
||||
"durationMs": 55,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-planes.test.js",
|
||||
"durationMs": 51,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-ip.test.js",
|
||||
"durationMs": 75,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-dest.test.js",
|
||||
"durationMs": 1836,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-classify.test.js",
|
||||
"durationMs": 903,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-ripe.test.js",
|
||||
"durationMs": 1440,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-brands.test.js",
|
||||
"durationMs": 93,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-ingest.test.js",
|
||||
"durationMs": 5340,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-analytics.test.js",
|
||||
"durationMs": 2172,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-map-hops.test.js",
|
||||
"durationMs": 1661,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-purge.test.js",
|
||||
"durationMs": 2451,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/geoip-download.test.js",
|
||||
"durationMs": 667,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-geoip.test.js",
|
||||
"durationMs": 1930,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-facts.test.js",
|
||||
"durationMs": 964,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-facts-filter.test.js",
|
||||
"durationMs": 1362,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/traffic-flow-facts-rebuild.test.js",
|
||||
"durationMs": 1833,
|
||||
"skips": []
|
||||
},
|
||||
{
|
||||
"path": "services/statistics-aggregate.test.js",
|
||||
"durationMs": 1915,
|
||||
"skips": []
|
||||
}
|
||||
]
|
||||
},
|
||||
"dedicatedIpv6": {
|
||||
"database": "mm_accept_144_1792139159368",
|
||||
"passed": true,
|
||||
"cases": [
|
||||
{
|
||||
"path": "services/traffic-flow-ipv6-pipeline.test.js",
|
||||
"durationMs": 7879,
|
||||
"skips": []
|
||||
}
|
||||
]
|
||||
},
|
||||
"dedicatedMonthly": {
|
||||
"database": "mm_accept_149_1792139159368",
|
||||
"passed": true,
|
||||
"cases": [
|
||||
{
|
||||
"path": "services/traffic-flow-month.test.js",
|
||||
"durationMs": 3800,
|
||||
"skips": []
|
||||
}
|
||||
]
|
||||
},
|
||||
"note": "The two explicitly guarded SQL cases skipped in primary were then executed without skips in separate owned databases. This profile predates only the runtime-counter route fix."
|
||||
},
|
||||
"live": {
|
||||
"sourceCommit": "db81cf0f",
|
||||
"passed": true,
|
||||
"checks": [
|
||||
{
|
||||
"name": "native-baselines-and-own-database-prepared",
|
||||
"routers": 2
|
||||
},
|
||||
{
|
||||
"name": "real-mikrotik-connected-1",
|
||||
"address": "198.18.254.2"
|
||||
},
|
||||
{
|
||||
"name": "real-mikrotik-connected-2",
|
||||
"address": "198.18.254.3"
|
||||
},
|
||||
{
|
||||
"name": "first-router-preserved-after-second",
|
||||
"peerCount": 3
|
||||
},
|
||||
{
|
||||
"name": "repeat-connect-without-duplicate"
|
||||
},
|
||||
{
|
||||
"name": "synthetic-ipfix-through-real-wireguard-to-postgresql",
|
||||
"packetsReceived": 4,
|
||||
"lastExporterIp": "198.18.254.200"
|
||||
},
|
||||
{
|
||||
"name": "agent-restart-preserves-tunnels-and-ipfix"
|
||||
},
|
||||
{
|
||||
"name": "backend-real-restart-reconciled",
|
||||
"peerCount": 4
|
||||
},
|
||||
{
|
||||
"name": "ipfix-after-backend-restart"
|
||||
},
|
||||
{
|
||||
"name": "persisted-exporter-summary",
|
||||
"exporters": [
|
||||
{
|
||||
"server_id": 1,
|
||||
"buckets": 44,
|
||||
"bytes": "18451"
|
||||
},
|
||||
{
|
||||
"server_id": 2,
|
||||
"buckets": 27,
|
||||
"bytes": "10281"
|
||||
},
|
||||
{
|
||||
"server_id": 3,
|
||||
"buckets": 1,
|
||||
"bytes": "432075"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "router-baseline-restored-2"
|
||||
},
|
||||
{
|
||||
"name": "router-baseline-restored-1"
|
||||
}
|
||||
],
|
||||
"nativeCleanup": true
|
||||
},
|
||||
"cleanup": {
|
||||
"nativeAndHostCleanup": true,
|
||||
"ownedDatabaseRemoved": true,
|
||||
"exitCode": 0
|
||||
},
|
||||
"finalVerification": {
|
||||
"passed": true,
|
||||
"services": [
|
||||
{
|
||||
"name": "mmapp-test-backend",
|
||||
"image": "mmapp-test/backend:ux-certificates258",
|
||||
"imageId": "sha256:984684709e619972bae8657208e3150d12f1c4f2b5642fecf5286a5d87e9fe71",
|
||||
"running": true,
|
||||
"health": "healthy",
|
||||
"imageUnchanged": true
|
||||
},
|
||||
{
|
||||
"name": "mmapp-test-frontend",
|
||||
"image": "mmapp-test/frontend:ux-accessibility261",
|
||||
"imageId": "sha256:eec4410f1109a46e864d695cfb4311fdb9a224247549c8a9e78e44c6c0e455cb",
|
||||
"running": true,
|
||||
"health": null,
|
||||
"imageUnchanged": true
|
||||
},
|
||||
{
|
||||
"name": "mmapp-test-portal",
|
||||
"image": "git.shx.one/denozord/auth-portal@sha256:2bdf42400a1b35caa43303b9d01078b054b2a9041eba7019c1fffde71b7e4489",
|
||||
"imageId": "sha256:2bdf42400a1b35caa43303b9d01078b054b2a9041eba7019c1fffde71b7e4489",
|
||||
"running": true,
|
||||
"health": "healthy",
|
||||
"imageUnchanged": true
|
||||
},
|
||||
{
|
||||
"name": "mmapp-test-postgres",
|
||||
"image": "postgres:18-alpine",
|
||||
"imageId": "sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873",
|
||||
"running": true,
|
||||
"health": "healthy",
|
||||
"imageUnchanged": true
|
||||
}
|
||||
],
|
||||
"temporaryContainersRemaining": 0,
|
||||
"hostOwnedWireguardRemaining": false,
|
||||
"privateFixturesRemoved": true,
|
||||
"mainFlowSettings": {
|
||||
"enabled": false,
|
||||
"collector_ip": "10.255.254.1",
|
||||
"prefix": "10.255.254.0/24",
|
||||
"flow_listen_port": 4739,
|
||||
"wg_listen_port": 51821,
|
||||
"peerCount": 0
|
||||
},
|
||||
"ownedDatabasesAndRolesRemaining": 0,
|
||||
"routers": [
|
||||
{
|
||||
"id": "1",
|
||||
"host": "194.226.121.150",
|
||||
"reachable": true,
|
||||
"ownedWireguardInterfaces": 0,
|
||||
"ownedFlowTargets": 0,
|
||||
"ownedCollectorRoutes": 0,
|
||||
"trafficFlowEnabled": "false"
|
||||
},
|
||||
{
|
||||
"id": "2",
|
||||
"host": "104.165.244.7",
|
||||
"reachable": true,
|
||||
"ownedWireguardInterfaces": 0,
|
||||
"ownedFlowTargets": 0,
|
||||
"ownedCollectorRoutes": 0,
|
||||
"trafficFlowEnabled": "false"
|
||||
}
|
||||
]
|
||||
},
|
||||
"correction": {
|
||||
"commit": "0abdac82",
|
||||
"problem": "Settings API returned stale persisted counters despite successful worker ingestion.",
|
||||
"change": "Read packet count, last exporter, error and packet time from collector runtime counters.",
|
||||
"regressionProof": "Initial live run persisted flows but timed out waiting for the settings packet counter after agent restart. Final run observed packetsReceived=4 and exporter 198.18.254.200, then passed both actual container restarts and continued reception.",
|
||||
"localValidation": [
|
||||
"Backend/contracts build",
|
||||
"8 netflow-settings test scripts",
|
||||
"ESLint for modified route"
|
||||
]
|
||||
},
|
||||
"earlierAttempts": [
|
||||
{
|
||||
"stage": "SQL runner",
|
||||
"result": "Harness FLOW_OPTIONAL truthiness corrected; primary profile rerun passed."
|
||||
},
|
||||
{
|
||||
"stage": "Live runner bootstrap",
|
||||
"result": "Production image lacks development pino-pretty; harness disabled pretty logging; resources cleaned."
|
||||
},
|
||||
{
|
||||
"stage": "Live counter verification",
|
||||
"result": "Product stale-counter defect detected and fixed; both routers and host restored before final full live rerun.",
|
||||
"failure": {
|
||||
"phase": "exercise",
|
||||
"name": "Error",
|
||||
"message": "Timed out: IPFIX continues during agent restart"
|
||||
}
|
||||
}
|
||||
],
|
||||
"nativeExportEvidence": "Own DB recorded server_id 1 and 2 through their distinct WG source addresses: 44 buckets/18451 bytes and 27 buckets/10281 bytes. Server 3 is the separate synthetic Linux IPFIX exporter (1 bucket/432075 bytes). These are exported flow-byte totals, not captured UDP wire bytes.",
|
||||
"limits": [
|
||||
"One-time installer was not applied to the primary stand deployment; its compose file is named compose.yaml. Installer integration with wg-quick/systemd and cold host reboot remain unverified on this stand.",
|
||||
"No production deployment, registry publication, or primary frontend/backend replacement was performed.",
|
||||
"Docker images and source/log artifacts are retained under /opt/mmapp-test/artifacts/netflow-agent-db81cf0f-20261016; temporary containers, owned DBs/roles and private fixture files were removed.",
|
||||
"This is a functional acceptance run, not a load or long-duration stability test."
|
||||
]
|
||||
}
|
||||
Reference in new issue
Block a user