feat(db): перевести хранилище с SQLite на PostgreSQL
Docker images / prepare-release (push) Successful in 12s
Docker images / backend-test (push) Successful in 4m9s
Docker images / frontend-image (push) Successful in 4m28s
Docker images / updater-image (push) Successful in 58s
Docker images / backend-image (push) Successful in 2m49s
Docker images / notify-webhook (push) Skipped
Docker images / publish-release (push) Successful in 11s

При старте backend накатывает схему PostgreSQL 18 и, если база пустая, один раз импортирует mikrotik.db с тома. Повторный старт не копирует данные. Бэкап в UI идёт через pg_dump.

Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Denozordec
2026-09-08 01:36:48 +07:00
co-authored by Cursor
parent 0e5fb065e2
commit ec43591a99
128 changed files with 4304 additions and 3639 deletions
+42 -1
View File
@@ -38,8 +38,49 @@ jobs:
node .ci/scripts/compute-release.mjs node .ci/scripts/compute-release.mjs
test -f .ci/release/release-manifest.json test -f .ci/release/release-manifest.json
backend-test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Node 22
uses: actions/setup-node@v4
with:
node-version: 22
- name: Start PostgreSQL 18
shell: bash
run: |
set -euo pipefail
docker run -d --name mm-pg \
-e POSTGRES_USER=mmapp \
-e POSTGRES_PASSWORD=mmapp \
-e POSTGRES_DB=mmapp \
-p 5432:5432 \
postgres:18-alpine
for i in $(seq 1 40); do
if docker exec mm-pg pg_isready -U mmapp -d mmapp; then
exit 0
fi
sleep 1
done
echo "PostgreSQL не поднялся"
exit 1
- name: Install and test backend
shell: bash
env:
DATABASE_URL: postgres://mmapp:[email protected]:5432/mmapp
run: |
set -euo pipefail
npm ci
npm run test --prefix backend
backend-image: backend-image:
needs: prepare-release needs:
- prepare-release
- backend-test
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Checkout - name: Checkout
+30 -21
View File
@@ -19,7 +19,7 @@
| Компонент | Путь | Стек | Порт (runtime) | Docker-образ | | Компонент | Путь | Стек | Порт (runtime) | Docker-образ |
|-----------|------|------|----------------|--------------| |-----------|------|------|----------------|--------------|
| Frontend | корень (`app/`, `components/`, `lib/`, …) | Next.js 16.2.4 (App Router), React 19 | 3000 | `…-frontend` | | Frontend | корень (`app/`, `components/`, `lib/`, …) | Next.js 16.2.4 (App Router), React 19 | 3000 | `…-frontend` |
| Backend | `backend/` | Fastify 5, TypeScript ESM, SQLite, Drizzle | 8000 | `…-backend` | | Backend | `backend/` | Fastify 5, TypeScript ESM, PostgreSQL 18, Drizzle | 8000 | `…-backend` |
| Контракты | `packages/contracts/` | Zod 4, `@mmapp/contracts` | — | встраиваются в frontend/backend | | Контракты | `packages/contracts/` | Zod 4, `@mmapp/contracts` | — | встраиваются в frontend/backend |
| Updater | `deploy/updater/` | bash, `docker:27-cli`, curl, jq | — | `…-updater` | | Updater | `deploy/updater/` | bash, `docker:27-cli`, curl, jq | — | `…-updater` |
| Деплой | `deploy/docker-compose.yml` | Docker Compose | — | — | | Деплой | `deploy/docker-compose.yml` | Docker Compose | — | — |
@@ -43,11 +43,11 @@ flowchart TB
Browser["Браузер"] Browser["Браузер"]
NextDev["Next.js :3000"] NextDev["Next.js :3000"]
FastifyDev["Fastify :8000"] FastifyDev["Fastify :8000"]
SqliteDev["SQLite файл"] PostgresDev["PostgreSQL 18"]
Contracts["@mmapp/contracts"] Contracts["@mmapp/contracts"]
Browser --> NextDev Browser --> NextDev
Browser -->|"fetch /api, CORS"| FastifyDev Browser -->|"fetch /api, CORS"| FastifyDev
FastifyDev --> SqliteDev FastifyDev --> PostgresDev
Contracts --> NextDev Contracts --> NextDev
Contracts --> FastifyDev Contracts --> FastifyDev
end end
@@ -60,7 +60,8 @@ flowchart TB
subgraph prod [Прод-сервер] subgraph prod [Прод-сервер]
FE["mmapp-frontend :3000"] FE["mmapp-frontend :3000"]
BE["mmapp-backend :8000"] BE["mmapp-backend :8000"]
DBVol["volume backend-data /app/data"] PG["mmapp-postgres"]
DBVol["volume sqlite ETL /app/data"]
UPD["mmapp-updater"] UPD["mmapp-updater"]
Sock["/var/run/docker.sock"] Sock["/var/run/docker.sock"]
Reg --> FE Reg --> FE
@@ -69,6 +70,7 @@ flowchart TB
UPD --> Sock UPD --> Sock
UPD --> FE UPD --> FE
UPD --> BE UPD --> BE
BE --> PG
BE --> DBVol BE --> DBVol
BrowserProd["Клиент"] --> FE BrowserProd["Клиент"] --> FE
BrowserProd -->|"live + backendUrl"| BE BrowserProd -->|"live + backendUrl"| BE
@@ -117,7 +119,7 @@ sequenceDiagram
``` ```
- Триггер workflow: `push` в ветку `main`, ручной `workflow_dispatch`. - Триггер workflow: `push` в ветку `main`, ручной `workflow_dispatch`.
- Параллельные jobs: `backend-image`, `frontend-image`, `updater-image`; опционально `notify-webhook` после backend и frontend, если задан секрет `DEPLOY_WEBHOOK_URL`. - Параллельные jobs: `backend-test` (PostgreSQL 18), `backend-image`, `frontend-image`, `updater-image`; опционально `notify-webhook` после backend и frontend, если задан секрет `DEPLOY_WEBHOOK_URL`.
- Теги на каждый успешный push: **`:latest`** и **`:<commit-sha>`**; платформа **linux/amd64**. - Теги на каждый успешный push: **`:latest`** и **`:<commit-sha>`**; платформа **linux/amd64**.
- На сервере образы подтягиваются вручную (`docker compose pull`) и/или через **updater** (сравнение digest у тега из `targets.json`). Webhook CI **не заменяет** updater. - На сервере образы подтягиваются вручную (`docker compose pull`) и/или через **updater** (сравнение digest у тега из `targets.json`). Webhook CI **не заменяет** updater.
@@ -125,7 +127,8 @@ sequenceDiagram
| Зависимость | Реализация | | Зависимость | Реализация |
|-------------|------------| |-------------|------------|
| SQLite | Не отдельный контейнер. Файл `mikrotik.db` в томе `backend-data``/app/data` (`DATABASE_PATH=/app/data/mikrotik.db` в образе backend). | | PostgreSQL | Контейнер `mmapp-postgres` (`postgres:18-alpine`). `DATABASE_URL=postgres://mmapp:…@postgres:5432/mmapp`. |
| SQLite (ETL) | Файл `mikrotik.db` на томе `/app/data` (`DATABASE_PATH=/app/data/mikrotik.db`). При первом старте, если PG пустой, backend сам импортирует данные и ставит маркер. Повторный старт не копирует заново. |
| Docker socket | Только у контейнера updater: `/var/run/docker.sock` — доступ к Docker API хоста (управление контейнерами, pull). | | Docker socket | Только у контейнера updater: `/var/run/docker.sock` — доступ к Docker API хоста (управление контейнерами, pull). |
## Локальная разработка ## Локальная разработка
@@ -134,7 +137,8 @@ sequenceDiagram
- **Node.js 22** (как в `Dockerfile.frontend` и `backend/Dockerfile`). - **Node.js 22** (как в `Dockerfile.frontend` и `backend/Dockerfile`).
- **npm** с workspaces; установка из корня: `npm ci` или `npm install`. - **npm** с workspaces; установка из корня: `npm ci` или `npm install`.
- Для нативной сборки `better-sqlite3` на Linux может понадобиться toolchain (`python3`, `make`, `g++`); в Docker-образе backend они уже ставятся. - Для нативной сборки `better-sqlite3` на Linux может понадобиться toolchain (`python3`, `make`, `g++`); в Docker-образе backend они уже ставятся (нужен только для одноразового boot-ETL из `mikrotik.db`).
- Локально нужен PostgreSQL 18 (`deploy/docker-compose.postgres.yml` или `DATABASE_URL`).
- Backend Docker-образ ставит только workspaces `backend` + `contracts` (без корневых Next/React deps); в production логи — JSON без `pino-pretty`. - Backend Docker-образ ставит только workspaces `backend` + `contracts` (без корневых Next/React deps); в production логи — JSON без `pino-pretty`.
### Запуск ### Запуск
@@ -161,7 +165,8 @@ Backend — скопировать `backend/.env.example` в `backend/.env`:
| Переменная | По умолчанию | Назначение | | Переменная | По умолчанию | Назначение |
|------------|--------------|------------| |------------|--------------|------------|
| `DATABASE_PATH` | `./mikrotik.db` | путь к файлу SQLite | | `DATABASE_URL` | `postgres://mmapp:[email protected]:5432/mmapp` | подключение к PostgreSQL |
| `DATABASE_PATH` | `./mikrotik.db` | SQLite только для boot-ETL, если PG пустой |
| `PORT` | `8000` | порт Fastify | | `PORT` | `8000` | порт Fastify |
| `CORS_ORIGIN` | `http://localhost:3000` | origin фронтенда для CORS | | `CORS_ORIGIN` | `http://localhost:3000` | origin фронтенда для CORS |
@@ -179,12 +184,10 @@ Frontend — в репозитории нет корневого `.env.example`.
npm run build -w @mmapp/contracts npm run build -w @mmapp/contracts
``` ```
После изменения схем Drizzle: Схема PostgreSQL накатывается при старте backend из [`backend/drizzle/0000_postgresql.sql`](backend/drizzle/0000_postgresql.sql). Опциональный ETL:
```bash ```bash
npm --prefix backend run db:generate npm --prefix backend run db:migrate-from-sqlite
npm --prefix backend run db:migrate
npm --prefix backend run db:studio
``` ```
## CI/CD (Gitea Actions) ## CI/CD (Gitea Actions)
@@ -281,7 +284,8 @@ SSO auth-portal: [`docs/integrate-auth-portal.md`](docs/integrate-auth-portal.md
| Сервис | `container_name` | Образ (пример) | Порты host:container | Тома | `restart` | | Сервис | `container_name` | Образ (пример) | Порты host:container | Тома | `restart` |
|--------|------------------|----------------|----------------------|------|-----------| |--------|------------------|----------------|----------------------|------|-----------|
| backend | `mmapp-backend` | `git.shx.one/denozord/mikrotikmanager-backend:latest` | `8000:8000` | `backend-data` `/app/data` | `unless-stopped` | | postgres | `mmapp-postgres` | `postgres:18-alpine` | нет (внутренняя сеть) | `mmapp-pgdata` | `unless-stopped` |
| backend | `mmapp-backend` | `git.shx.one/denozord/mikrotikmanager-backend:latest` | `8000:8000` | sqlite ETL → `/app/data` | `unless-stopped` |
| frontend | `mmapp-frontend` | `git.shx.one/denozord/mikrotikmanager-frontend:latest` | `3000:3000` | — | `unless-stopped` | | frontend | `mmapp-frontend` | `git.shx.one/denozord/mikrotikmanager-frontend:latest` | `3000:3000` | — | `unless-stopped` |
| updater | `mmapp-updater` | `git.shx.one/denozord/mikrotikmanager-updater:latest` | не публикуются | docker.sock, `updater-state``/state`, `targets.json``/etc/updater/targets.json:ro` | `unless-stopped` | | updater | `mmapp-updater` | `git.shx.one/denozord/mikrotikmanager-updater:latest` | не публикуются | docker.sock, `updater-state``/state`, `targets.json``/etc/updater/targets.json:ro` | `unless-stopped` |
@@ -297,11 +301,14 @@ SSO auth-portal: [`docs/integrate-auth-portal.md`](docs/integrate-auth-portal.md
### Переменные окружения (прод) ### Переменные окружения (прод)
**Backend** (в образе заданы `NODE_ENV=production`, `PORT=8000`, `DATABASE_PATH=/app/data/mikrotik.db`; в compose обычно переопределяют только CORS): **Backend** (в образе заданы `NODE_ENV=production`, `PORT=8000`; в compose задают `DATABASE_URL`, `DATABASE_PATH` для ETL и CORS):
| Переменная | Источник в compose | Назначение | | Переменная | Источник в compose | Назначение |
|------------|-------------------|------------| |------------|-------------------|------------|
| `CORS_ORIGIN` | `${CORS_ORIGIN:-http://localhost:3000}` | origin UI, с которого браузер вызывает API | | `CORS_ORIGIN` | `${CORS_ORIGIN:-http://localhost:3000}` | origin UI, с которого браузер вызывает API |
| `DATABASE_URL` | `postgres://mmapp:${POSTGRES_PASSWORD}@postgres:5432/mmapp` | PostgreSQL |
| `DATABASE_PATH` | `/app/data/mikrotik.db` | SQLite на томе для одноразового ETL |
| `POSTGRES_PASSWORD` | обязателен | пароль пользователя `mmapp` |
**Frontend** (в образе: `PORT=3000`, `HOSTNAME=0.0.0.0`). URL API в образ **не** зашит — задаётся в браузере (режим live + URL бэкенда) вместе с `CORS_ORIGIN` на backend. **Frontend** (в образе: `PORT=3000`, `HOSTNAME=0.0.0.0`). URL API в образ **не** зашит — задаётся в браузере (режим live + URL бэкенда) вместе с `CORS_ORIGIN` на backend.
@@ -312,7 +319,7 @@ SSO auth-portal: [`docs/integrate-auth-portal.md`](docs/integrate-auth-portal.md
| `TARGETS_FILE` | `/etc/updater/targets.json` | список целей | | `TARGETS_FILE` | `/etc/updater/targets.json` | список целей |
| `STATE_FILE` | `/state/updater-state.json` | digest и ошибки | | `STATE_FILE` | `/state/updater-state.json` | digest и ошибки |
| `POLL_INTERVAL_SECONDS` | `300` | пауза между циклами опроса | | `POLL_INTERVAL_SECONDS` | `300` | пауза между циклами опроса |
| `HEALTH_TIMEOUT_SECONDS` | `120` | ожидание HTTP health | | `HEALTH_TIMEOUT_SECONDS` | `600` | ожидание HTTP health (первый boot с ETL может занять минуты) |
| `STOP_TIMEOUT_SECONDS` | `30` | `docker stop -t` | | `STOP_TIMEOUT_SECONDS` | `30` | `docker stop -t` |
| `REGISTRY` | `git.shx.one` | registry для `docker login` | | `REGISTRY` | `git.shx.one` | registry для `docker login` |
| `REGISTRY_USERNAME` | пусто | логин (если заданы оба с паролем) | | `REGISTRY_USERNAME` | пусто | логин (если заданы оба с паролем) |
@@ -516,18 +523,20 @@ docker compose up -d --force-recreate
### Откат ### Откат
- **Автоматически:** updater при failed health после обновления — контейнер на `previous_digest`. - **Автоматически:** updater при failed health после обновления — контейнер на `previous_digest`.
- **Вручную:** остановить контейнер, запустить образ с нужным тегом или digest из registry, сохранив те же volume и labels. Откат **схемы SQLite** updater не выполняет — нужен отдельный backup тома `backend-data` / файла `mikrotik.db`. - **Вручную:** остановить контейнер, запустить образ с нужным тегом или digest из registry, сохранив те же volume и labels. Откат **схемы PostgreSQL** updater не выполняет — нужен `pg_dump` / restore или том `mmapp-pgdata`.
### Резервное копирование SQLite ### Резервное копирование PostgreSQL
Том `backend-data` (или `mmapp-backend-data` при явном `docker volume create`). Пример остановки backend для консистентной копии: Том `mmapp-pgdata` и (до cleanup) sqlite `mikrotik.db` на `/app/data`. Из UI: Настройки → «Скачать бэкап» (`pg_dump -Fc`). С хоста:
```bash ```bash
docker stop mmapp-backend docker exec mmapp-postgres pg_dump -Fc -U mmapp mmapp > mikrotik-manager.dump
docker run --rm -v backend-data:/data -v $(pwd):/backup alpine tar czf /backup/mikrotik-db-backup.tar.gz -C /data .
docker start mmapp-backend
``` ```
Восстановление: UI «Восстановить из файла» или `pg_restore --clean --if-exists -d …`.
Том sqlite после успешного ETL и smoke **не удаляйте сразу** — оставьте как fallback, затем уберите `mikrotik.db`, когда маркер `data_migration.sqlite_imported_at` подтверждён.
### Отладка ### Отладка
| Действие | Команда | | Действие | Команда |
+1 -1
View File
@@ -2846,7 +2846,7 @@ export default function AlertsPage() {
</CollapsibleTrigger> </CollapsibleTrigger>
<CollapsibleContent className="px-3 pb-3 text-[11px] text-muted-foreground leading-relaxed space-y-2 border-t border-border/60 pt-2"> <CollapsibleContent className="px-3 pb-3 text-[11px] text-muted-foreground leading-relaxed space-y-2 border-t border-border/60 pt-2">
<p> <p>
Движок читает последние сэмплы из SQLite (uptime ресурсы, REST-пинг, пробы, GRE/BGP). Условие Движок читает последние сэмплы из PostgreSQL (uptime ресурсы, REST-пинг, пробы, GRE/BGP). Условие
срабатывает только если цепочка состояний совпадает с текстом правила. срабатывает только если цепочка состояний совпадает с текстом правила.
</p> </p>
<ul className="list-disc pl-4 space-y-1.5"> <ul className="list-disc pl-4 space-y-1.5">
+3 -3
View File
@@ -220,7 +220,7 @@ function SnapshotTables({ snap }: { snap: SchedulerRunSnapshot }) {
</Alert> </Alert>
) : null} ) : null}
<p className="text-xs text-muted-foreground"> <p className="text-xs text-muted-foreground">
Запись в SQLite на{" "} Запись в PostgreSQL на{" "}
<span className="font-mono tabular-nums">{new Date(g.sampledAt).toLocaleString("ru-RU")}</span> <span className="font-mono tabular-nums">{new Date(g.sampledAt).toLocaleString("ru-RU")}</span>
{" "} строки GRE и BGP для движка оповещений. {" "} строки GRE и BGP для движка оповещений.
</p> </p>
@@ -347,7 +347,7 @@ function SnapshotTables({ snap }: { snap: SchedulerRunSnapshot }) {
<p className="text-xs text-muted-foreground"> <p className="text-xs text-muted-foreground">
Снимок на{" "} Снимок на{" "}
<span className="font-mono tabular-nums">{new Date(a.sampledAt).toLocaleString("ru-RU")}</span> <span className="font-mono tabular-nums">{new Date(a.sampledAt).toLocaleString("ru-RU")}</span>
. Данные для правил берутся из SQLite после джоб сбора (трафик, uptime, REST, GRE+BGP и т.д.). . Данные для правил берутся из PostgreSQL после джоб сбора (трафик, uptime, REST, GRE+BGP и т.д.).
</p> </p>
<dl className="grid grid-cols-2 sm:grid-cols-4 gap-3 text-xs"> <dl className="grid grid-cols-2 sm:grid-cols-4 gap-3 text-xs">
<div> <div>
@@ -1215,7 +1215,7 @@ export default function DataCollectionPage() {
<OpsPanel <OpsPanel
title="Журнал прогонов" title="Журнал прогонов"
description="SQLite `scheduler_runs` — до 80 записей; раскройте строку для полей и текста ошибки." description="Таблица `scheduler_runs` — до 80 записей; раскройте строку для полей и текста ошибки."
headerRight={ headerRight={
<div className="flex items-center gap-2 shrink-0"> <div className="flex items-center gap-2 shrink-0">
<label className="text-xs text-muted-foreground whitespace-nowrap" htmlFor="run-filter"> <label className="text-xs text-muted-foreground whitespace-nowrap" htmlFor="run-filter">
+8 -8
View File
@@ -88,7 +88,7 @@ function DatabaseRestoreConfirm({
</AlertDialogMedia> </AlertDialogMedia>
<AlertDialogTitle>Восстановить базу приложения?</AlertDialogTitle> <AlertDialogTitle>Восстановить базу приложения?</AlertDialogTitle>
<AlertDialogDescription> <AlertDialogDescription>
Файл: {filename}. Текущие данные SQLite на бекенде будут полностью заменены. Файл: {filename}. Текущие данные PostgreSQL на бекенде будут полностью заменены.
Рекомендуется сначала скачать актуальный бэкап. Рекомендуется сначала скачать актуальный бэкап.
</AlertDialogDescription> </AlertDialogDescription>
</AlertDialogHeader> </AlertDialogHeader>
@@ -435,7 +435,7 @@ export default function SettingsPage() {
<OpsPanel <OpsPanel
title="База данных приложения" title="База данных приложения"
description="Резервная копия SQLite бекенда и сброс таблиц NetFlow. На время операции планировщик и коллектор IPFIX приостанавливаются. Preview: https://reui.io/preview/base/settings-16" description="Резервная копия PostgreSQL (pg_dump custom) и сброс таблиц NetFlow. На время операции планировщик и коллектор IPFIX приостанавливаются. Preview: https://reui.io/preview/base/settings-16"
contentClassName="divide-y px-5" contentClassName="divide-y px-5"
> >
{!systemDbAvailable && ( {!systemDbAvailable && (
@@ -446,7 +446,7 @@ export default function SettingsPage() {
)} )}
<SettingRow <SettingRow
label="Скачать бэкап" label="Скачать бэкап"
description="Консистентная копия файла mikrotik.db" description="Консистентный дамп PostgreSQL (pg_dump -Fc)"
> >
<Button <Button
variant="outline" variant="outline"
@@ -461,7 +461,7 @@ export default function SettingsPage() {
</SettingRow> </SettingRow>
<SettingRow <SettingRow
label="Восстановить из файла" label="Восстановить из файла"
description="Полностью заменяет текущую базу SQLite" description="Полностью заменяет текущую базу PostgreSQL (pg_restore)"
> >
<div className="flex flex-col items-end gap-2"> <div className="flex flex-col items-end gap-2">
<Button <Button
@@ -481,7 +481,7 @@ export default function SettingsPage() {
</SettingRow> </SettingRow>
<SettingRow <SettingRow
label="Сбросить данные NetFlow" label="Сбросить данные NetFlow"
description="Удалит сессии и агрегаты из SQLite, затем VACUUM. Ключи WG и пиры не трогает" description="Удалит сессии и агрегаты NetFlow в PostgreSQL. Ключи WG и пиры не трогает"
> >
<Button <Button
variant="destructive" variant="destructive"
@@ -563,7 +563,7 @@ export default function SettingsPage() {
> >
access.md access.md
</a> </a>
. URL и ключ хранятся в SQLite на сервере бекенда ( . URL и ключ хранятся в PostgreSQL на сервере бекенда (
<code className="font-mono bg-muted px-1 rounded">evobgp_settings</code> <code className="font-mono bg-muted px-1 rounded">evobgp_settings</code>
). Каталог {" "} ). Каталог {" "}
<code className="font-mono bg-muted px-1 rounded">GET /v1/router-lists/catalog</code> через прокси. <code className="font-mono bg-muted px-1 rounded">GET /v1/router-lists/catalog</code> через прокси.
@@ -937,8 +937,8 @@ export default function SettingsPage() {
open={dbRestoreDialogOpen} open={dbRestoreDialogOpen}
onOpenChange={setDbRestoreDialogOpen} onOpenChange={setDbRestoreDialogOpen}
title="Восстановление базы данных" title="Восстановление базы данных"
description="Выберите файл SQLite (.db) — текущая база будет полностью заменена" description="Выберите файл pg_dump custom (.dump) — текущая база будет полностью заменена"
accept=".db,.sqlite,.sqlite3,application/octet-stream" accept=".dump,.backup,application/octet-stream"
onImport={async (files) => { onImport={async (files) => {
const file = files[0] const file = files[0]
if (!file) return if (!file) return
+5 -1
View File
@@ -1,5 +1,9 @@
# Path to SQLite database file # PostgreSQL connection string
DATABASE_URL=postgres://mmapp:[email protected]:5432/mmapp
# SQLite file used only for one-shot boot ETL (empty PG + existing mikrotik.db)
DATABASE_PATH=./mikrotik.db DATABASE_PATH=./mikrotik.db
# SQLITE_IMPORT_FULL_HISTORY=true
# Port for the Fastify server # Port for the Fastify server
PORT=8000 PORT=8000
+9 -1
View File
@@ -47,8 +47,16 @@ COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/packages/contracts ./packages/contracts COPY --from=build /app/packages/contracts ./packages/contracts
COPY --from=build /app/backend/dist ./backend/dist COPY --from=build /app/backend/dist ./backend/dist
COPY --from=build /app/backend/package.json ./backend/package.json COPY --from=build /app/backend/package.json ./backend/package.json
COPY --from=build /app/backend/drizzle ./backend/drizzle
# Nested install from lockfile (dotenv etc.) — ESM resolves from /app/backend/dist → ../node_modules # Nested install from lockfile (dotenv etc.) — ESM resolves from /app/backend/dist → ../node_modules
COPY --from=build /app/backend/node_modules ./backend/node_modules COPY --from=build /app/backend/node_modules ./backend/node_modules
RUN mkdir -p /app/data RUN apt-get update \
&& apt-get install -y --no-install-recommends gnupg wget ca-certificates \
&& wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/pgdg.gpg \
&& echo "deb [signed-by=/usr/share/keyrings/pgdg.gpg] http://apt.postgresql.org/pub/repos/apt bookworm-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends postgresql-client-18 \
&& rm -rf /var/lib/apt/lists/* \
&& mkdir -p /app/data
EXPOSE 8000 EXPOSE 8000
CMD ["node", "backend/dist/index.js"] CMD ["node", "backend/dist/index.js"]
+2 -2
View File
@@ -6,9 +6,9 @@ config()
export default defineConfig({ export default defineConfig({
schema: "./src/db/schema.ts", schema: "./src/db/schema.ts",
out: "./drizzle", out: "./drizzle",
dialect: "sqlite", dialect: "postgresql",
dbCredentials: { dbCredentials: {
url: process.env.DATABASE_PATH ?? "./mikrotik.db", url: process.env.DATABASE_URL ?? "postgres://mmapp:[email protected]:5432/mmapp",
}, },
verbose: true, verbose: true,
strict: true, strict: true,
-36
View File
@@ -1,36 +0,0 @@
CREATE TABLE `server_snapshots` (
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
`server_id` integer NOT NULL,
`polled_at` text NOT NULL,
`status` text NOT NULL,
`latency_ms` real,
`ros_version` text,
`board_name` text,
`uptime` text,
`cpu_load` integer,
`free_memory` integer,
`total_memory` integer,
`identity_name` text,
`raw_interfaces` text,
`raw_ip_addresses` text,
FOREIGN KEY (`server_id`) REFERENCES `servers`(`id`) ON UPDATE no action ON DELETE cascade
);
--> statement-breakpoint
CREATE TABLE `servers` (
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
`name` text DEFAULT '' NOT NULL,
`host` text NOT NULL,
`port` integer DEFAULT 443 NOT NULL,
`username` text DEFAULT 'admin' NOT NULL,
`password` text DEFAULT '' NOT NULL,
`use_ssl` integer DEFAULT true NOT NULL,
`verify_ssl` integer DEFAULT false NOT NULL,
`type` text DEFAULT 'home-router' NOT NULL,
`site` text DEFAULT '' NOT NULL,
`country` text DEFAULT '' NOT NULL,
`asn` text DEFAULT '' NOT NULL,
`comment` text DEFAULT '' NOT NULL,
`enabled` integer DEFAULT true NOT NULL,
`created_at` text DEFAULT (datetime('now')) NOT NULL,
`updated_at` text DEFAULT (datetime('now')) NOT NULL
);
+626
View File
@@ -0,0 +1,626 @@
-- MikrotikManager PostgreSQL 18 schema (not 1:1 SQLite).
-- Applied once at backend boot via src/db/migrate.ts
CREATE TABLE IF NOT EXISTS schema_migrations (
id TEXT PRIMARY KEY,
applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS servers (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
name TEXT NOT NULL DEFAULT '',
host TEXT NOT NULL,
port INTEGER NOT NULL DEFAULT 443,
username TEXT NOT NULL DEFAULT 'admin',
password TEXT NOT NULL DEFAULT '',
use_ssl BOOLEAN NOT NULL DEFAULT TRUE,
verify_ssl BOOLEAN NOT NULL DEFAULT FALSE,
type TEXT NOT NULL DEFAULT 'home-router'
CHECK (type IN ('jump-host', 'exit-node', 'home-router')),
site TEXT NOT NULL DEFAULT '',
country TEXT NOT NULL DEFAULT '',
asn TEXT NOT NULL DEFAULT '',
comment TEXT NOT NULL DEFAULT '',
enabled BOOLEAN NOT NULL DEFAULT TRUE,
lan_subnet TEXT NOT NULL DEFAULT '',
wan_uplinks JSONB NOT NULL DEFAULT '[]'::jsonb,
mgmt_tunnel_ip TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS traffic_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
enabled BOOLEAN NOT NULL DEFAULT TRUE,
interval_sec INTEGER NOT NULL DEFAULT 30,
retention_days INTEGER NOT NULL DEFAULT 14,
last_collected_at TIMESTAMPTZ,
last_duration_ms INTEGER,
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS servers_api_ping_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
enabled BOOLEAN NOT NULL DEFAULT FALSE,
interval_sec INTEGER NOT NULL DEFAULT 120,
last_collected_at TIMESTAMPTZ,
last_duration_ms INTEGER,
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS traffic_flow_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
enabled BOOLEAN NOT NULL DEFAULT FALSE,
collector_ip TEXT NOT NULL DEFAULT '10.255.254.1',
flow_listen_port INTEGER NOT NULL DEFAULT 4739,
wg_listen_port INTEGER NOT NULL DEFAULT 51821,
prefix TEXT NOT NULL DEFAULT '10.255.254.0/24',
public_endpoint TEXT NOT NULL DEFAULT '',
host_public_key TEXT NOT NULL DEFAULT '',
host_private_key TEXT NOT NULL DEFAULT '',
hub_server_id BIGINT,
retention_hours INTEGER NOT NULL DEFAULT 24,
top_n INTEGER NOT NULL DEFAULT 200,
map_service_min_share_pct DOUBLE PRECISION NOT NULL DEFAULT 5,
last_datagram_at TIMESTAMPTZ,
last_exporter_ip TEXT,
last_error TEXT,
packets_received BIGINT NOT NULL DEFAULT 0,
peers_json JSONB NOT NULL DEFAULT '[]'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS uptime_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
enabled BOOLEAN NOT NULL DEFAULT TRUE,
resources_enabled BOOLEAN NOT NULL DEFAULT TRUE,
ping_enabled BOOLEAN NOT NULL DEFAULT TRUE,
speed_enabled BOOLEAN NOT NULL DEFAULT TRUE,
interval_sec INTEGER NOT NULL DEFAULT 15,
probe_interval_sec INTEGER NOT NULL DEFAULT 15,
speed_interval_sec INTEGER NOT NULL DEFAULT 60,
retention_days INTEGER NOT NULL DEFAULT 14,
last_collected_at TIMESTAMPTZ,
last_duration_ms INTEGER,
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS evobgp_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
base_url TEXT NOT NULL DEFAULT '',
api_key TEXT NOT NULL DEFAULT '',
enabled BOOLEAN NOT NULL DEFAULT FALSE,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS alert_telegram_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
bot_token TEXT NOT NULL DEFAULT '',
chat_id TEXT NOT NULL DEFAULT '',
message_thread_id INTEGER,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS acme_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
directory_url TEXT NOT NULL DEFAULT 'https://acme-v02.api.letsencrypt.org/directory',
cloudflare_api_token TEXT NOT NULL DEFAULT '',
default_zone_id TEXT NOT NULL DEFAULT '',
account_private_key TEXT NOT NULL DEFAULT '',
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS certificate_renew_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
enabled BOOLEAN NOT NULL DEFAULT TRUE,
interval_sec INTEGER NOT NULL DEFAULT 21600,
renew_before_days INTEGER NOT NULL DEFAULT 30,
last_collected_at TIMESTAMPTZ,
last_duration_ms INTEGER,
last_error TEXT,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS backup_schedule_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
enabled BOOLEAN NOT NULL DEFAULT TRUE,
frequency TEXT NOT NULL DEFAULT 'daily' CHECK (frequency IN ('daily', 'weekly', 'monthly')),
hour INTEGER NOT NULL DEFAULT 3,
minute INTEGER NOT NULL DEFAULT 0,
week_day INTEGER NOT NULL DEFAULT 0,
month_day INTEGER NOT NULL DEFAULT 1,
keep_count INTEGER NOT NULL DEFAULT 7,
format TEXT NOT NULL DEFAULT 'rsc' CHECK (format IN ('rsc', 'backup')),
server_ids_json JSONB NOT NULL DEFAULT '[]'::jsonb,
last_run_at TIMESTAMPTZ,
last_duration_ms INTEGER,
last_error TEXT,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS internet_path_settings (
id BIGINT PRIMARY KEY CHECK (id = 1),
enabled BOOLEAN NOT NULL DEFAULT TRUE,
interval_sec INTEGER NOT NULL DEFAULT 300,
retention_days INTEGER NOT NULL DEFAULT 14,
last_collected_at TIMESTAMPTZ,
last_duration_ms INTEGER,
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS alert_engine_cursor (
id BIGINT PRIMARY KEY CHECK (id = 1),
last_source_finished_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS data_migration (
id INTEGER PRIMARY KEY CHECK (id = 1),
sqlite_imported_at TIMESTAMPTZ,
sqlite_path TEXT,
sqlite_sha256 TEXT,
report_json JSONB
);
CREATE TABLE IF NOT EXISTS filter_rules (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
sort_order INTEGER NOT NULL DEFAULT 0,
community TEXT NOT NULL,
community_name TEXT,
action TEXT NOT NULL DEFAULT 'route' CHECK (action IN ('route', 'blackhole')),
gateway TEXT NOT NULL DEFAULT '',
gateway_tunnel_id TEXT NOT NULL DEFAULT '',
description TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_filter_rules_server_sort ON filter_rules(server_id, sort_order);
CREATE TABLE IF NOT EXISTS recursive_routes (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
sort_order INTEGER NOT NULL DEFAULT 0,
dst_address TEXT NOT NULL,
gateway TEXT NOT NULL,
distance INTEGER NOT NULL DEFAULT 1,
scope INTEGER,
target_scope INTEGER,
routing_table TEXT NOT NULL DEFAULT 'main',
check_gateway TEXT NOT NULL DEFAULT '',
country TEXT NOT NULL DEFAULT '',
comment TEXT NOT NULL DEFAULT '',
disabled BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_recursive_routes_server_sort ON recursive_routes(server_id, sort_order);
CREATE TABLE IF NOT EXISTS server_snapshots (
id BIGINT GENERATED BY DEFAULT AS IDENTITY,
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
polled_at TIMESTAMPTZ NOT NULL,
status TEXT NOT NULL CHECK (status IN ('online', 'offline')),
latency_ms DOUBLE PRECISION,
ros_version TEXT,
board_name TEXT,
uptime TEXT,
cpu_load INTEGER,
free_memory BIGINT,
total_memory BIGINT,
identity_name TEXT,
raw_interfaces JSONB,
raw_ip_addresses JSONB,
PRIMARY KEY (id, polled_at)
) PARTITION BY RANGE (polled_at);
CREATE INDEX IF NOT EXISTS idx_server_snapshots_server_time ON server_snapshots(server_id, polled_at);
CREATE INDEX IF NOT EXISTS idx_server_snapshots_polled_brin ON server_snapshots USING BRIN (polled_at);
CREATE TABLE IF NOT EXISTS traffic_samples (
id BIGINT GENERATED BY DEFAULT AS IDENTITY,
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
interface_name TEXT NOT NULL,
peer_public_key TEXT NOT NULL DEFAULT '',
sampled_at TIMESTAMPTZ NOT NULL,
rx_bytes BIGINT NOT NULL DEFAULT 0,
tx_bytes BIGINT NOT NULL DEFAULT 0,
rx_bps BIGINT NOT NULL DEFAULT 0,
tx_bps BIGINT NOT NULL DEFAULT 0,
running BOOLEAN NOT NULL DEFAULT FALSE,
disabled BOOLEAN NOT NULL DEFAULT FALSE,
PRIMARY KEY (id, sampled_at)
) PARTITION BY RANGE (sampled_at);
CREATE INDEX IF NOT EXISTS idx_traffic_samples_server_time ON traffic_samples(server_id, sampled_at);
CREATE INDEX IF NOT EXISTS idx_traffic_samples_server_iface_time ON traffic_samples(server_id, interface_name, sampled_at);
CREATE INDEX IF NOT EXISTS idx_traffic_samples_sampled_brin ON traffic_samples USING BRIN (sampled_at);
CREATE TABLE IF NOT EXISTS servers_rest_ping_samples (
id BIGINT GENERATED BY DEFAULT AS IDENTITY,
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
sampled_at TIMESTAMPTZ NOT NULL,
ok BOOLEAN NOT NULL,
latency_ms INTEGER,
error TEXT,
PRIMARY KEY (id, sampled_at)
) PARTITION BY RANGE (sampled_at);
CREATE INDEX IF NOT EXISTS idx_servers_rest_ping_samples_server_id ON servers_rest_ping_samples(server_id, sampled_at);
CREATE INDEX IF NOT EXISTS idx_servers_rest_ping_samples_brin ON servers_rest_ping_samples USING BRIN (sampled_at);
CREATE TABLE IF NOT EXISTS flow_buckets (
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
bucket_at TIMESTAMPTZ NOT NULL,
src TEXT NOT NULL,
dst TEXT NOT NULL,
proto INTEGER NOT NULL DEFAULT 0,
src_port INTEGER NOT NULL DEFAULT 0,
dst_port INTEGER NOT NULL DEFAULT 0,
bytes BIGINT NOT NULL DEFAULT 0,
packets BIGINT NOT NULL DEFAULT 0,
in_iface TEXT NOT NULL DEFAULT '',
out_iface TEXT NOT NULL DEFAULT '',
next_hop TEXT NOT NULL DEFAULT '',
flow_start_ms BIGINT NOT NULL DEFAULT 0,
flow_end_ms BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (server_id, bucket_at, src, dst, proto, src_port, dst_port, in_iface)
) PARTITION BY RANGE (bucket_at);
CREATE INDEX IF NOT EXISTS idx_flow_buckets_server_time ON flow_buckets(server_id, bucket_at);
CREATE INDEX IF NOT EXISTS idx_flow_buckets_bucket_brin ON flow_buckets USING BRIN (bucket_at);
CREATE TABLE IF NOT EXISTS flow_minute_stats (
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
bucket_at TIMESTAMPTZ NOT NULL,
bytes BIGINT NOT NULL DEFAULT 0,
packets BIGINT NOT NULL DEFAULT 0,
unique_src INTEGER NOT NULL DEFAULT 0,
unique_dst INTEGER NOT NULL DEFAULT 0,
conversations INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (server_id, bucket_at)
) PARTITION BY RANGE (bucket_at);
CREATE INDEX IF NOT EXISTS idx_flow_minute_stats_brin ON flow_minute_stats USING BRIN (bucket_at);
CREATE TABLE IF NOT EXISTS flow_minute_dims (
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
bucket_at TIMESTAMPTZ NOT NULL,
dim TEXT NOT NULL,
key TEXT NOT NULL,
bytes BIGINT NOT NULL DEFAULT 0,
packets BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (server_id, bucket_at, dim, key)
) PARTITION BY RANGE (bucket_at);
CREATE INDEX IF NOT EXISTS idx_flow_minute_dims_time ON flow_minute_dims(bucket_at, dim);
CREATE INDEX IF NOT EXISTS idx_flow_minute_dims_brin ON flow_minute_dims USING BRIN (bucket_at);
CREATE TABLE IF NOT EXISTS flow_daily_dims (
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
day DATE NOT NULL,
dim TEXT NOT NULL,
key TEXT NOT NULL,
bytes BIGINT NOT NULL DEFAULT 0,
packets BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (server_id, day, dim, key)
) PARTITION BY RANGE (day);
CREATE INDEX IF NOT EXISTS idx_flow_daily_dims_day ON flow_daily_dims(day, dim);
CREATE TABLE IF NOT EXISTS flow_ip_meta (
prefix TEXT PRIMARY KEY,
asn INTEGER NOT NULL DEFAULT 0,
country TEXT NOT NULL DEFAULT '',
lat DOUBLE PRECISION,
lng DOUBLE PRECISION,
holder TEXT NOT NULL DEFAULT '',
ok INTEGER NOT NULL DEFAULT 1,
fetched_at TIMESTAMPTZ NOT NULL
);
CREATE TABLE IF NOT EXISTS flow_asn_meta (
asn INTEGER PRIMARY KEY,
holder TEXT NOT NULL DEFAULT '',
fetched_at TIMESTAMPTZ NOT NULL
);
CREATE TABLE IF NOT EXISTS uptime_probes (
id TEXT PRIMARY KEY,
src_server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
src_interface TEXT NOT NULL DEFAULT '',
name TEXT NOT NULL,
target TEXT NOT NULL,
probe_filter TEXT NOT NULL DEFAULT '',
enabled BOOLEAN NOT NULL DEFAULT TRUE,
interval_sec INTEGER NOT NULL DEFAULT 0,
show_on_dashboard BOOLEAN NOT NULL DEFAULT FALSE,
sort_order INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_uptime_probes_server_sort ON uptime_probes(src_server_id, sort_order);
CREATE TABLE IF NOT EXISTS uptime_probe_samples (
id BIGINT GENERATED BY DEFAULT AS IDENTITY,
probe_id TEXT NOT NULL REFERENCES uptime_probes(id) ON DELETE CASCADE,
sampled_at TIMESTAMPTZ NOT NULL,
rtt_ms INTEGER,
loss_pct INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'down' CHECK (status IN ('up', 'warn', 'down')),
PRIMARY KEY (id, sampled_at)
) PARTITION BY RANGE (sampled_at);
CREATE INDEX IF NOT EXISTS idx_uptime_probe_samples_probe_time ON uptime_probe_samples(probe_id, sampled_at);
CREATE INDEX IF NOT EXISTS idx_uptime_probe_samples_brin ON uptime_probe_samples USING BRIN (sampled_at);
CREATE TABLE IF NOT EXISTS uptime_resource_samples (
id BIGINT GENERATED BY DEFAULT AS IDENTITY,
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
sampled_at TIMESTAMPTZ NOT NULL,
status TEXT NOT NULL DEFAULT 'offline' CHECK (status IN ('online', 'offline')),
cpu_load INTEGER NOT NULL DEFAULT 0,
free_memory BIGINT NOT NULL DEFAULT 0,
total_memory BIGINT NOT NULL DEFAULT 0,
free_hdd_space BIGINT NOT NULL DEFAULT 0,
total_hdd_space BIGINT NOT NULL DEFAULT 0,
uptime_seconds BIGINT NOT NULL DEFAULT 0,
board_name TEXT NOT NULL DEFAULT '',
ros_version TEXT NOT NULL DEFAULT '',
PRIMARY KEY (id, sampled_at)
) PARTITION BY RANGE (sampled_at);
CREATE INDEX IF NOT EXISTS idx_uptime_resource_samples_server_time ON uptime_resource_samples(server_id, sampled_at);
CREATE INDEX IF NOT EXISTS idx_uptime_resource_samples_brin ON uptime_resource_samples USING BRIN (sampled_at);
CREATE TABLE IF NOT EXISTS uptime_speed_probes (
id TEXT PRIMARY KEY,
src_server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
dst_server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
src_interface TEXT NOT NULL DEFAULT '',
dst_interface TEXT NOT NULL DEFAULT '',
protocol TEXT NOT NULL DEFAULT 'tcp' CHECK (protocol IN ('tcp', 'udp')),
direction TEXT NOT NULL DEFAULT 'both' CHECK (direction IN ('transmit', 'receive', 'both')),
duration_sec INTEGER NOT NULL DEFAULT 10,
enabled BOOLEAN NOT NULL DEFAULT TRUE,
last_run_at TIMESTAMPTZ,
last_tx_avg_mbps DOUBLE PRECISION,
last_rx_avg_mbps DOUBLE PRECISION,
last_status TEXT CHECK (last_status IN ('done', 'error')),
last_error TEXT,
last_ping_rtt_ms INTEGER,
last_ping_loss_pct INTEGER,
last_ping_at TIMESTAMPTZ,
last_ping_error TEXT,
sort_order INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_uptime_speed_probes_src_sort ON uptime_speed_probes(src_server_id, sort_order);
CREATE TABLE IF NOT EXISTS uptime_speed_test_runs (
id TEXT PRIMARY KEY,
probe_id TEXT,
src_server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
dst_server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
src_interface TEXT NOT NULL DEFAULT '',
dst_interface TEXT NOT NULL DEFAULT '',
src_address TEXT,
dst_address TEXT,
src_interface_address TEXT,
dst_interface_address TEXT,
protocol TEXT NOT NULL DEFAULT 'tcp' CHECK (protocol IN ('tcp', 'udp')),
direction TEXT NOT NULL DEFAULT 'both' CHECK (direction IN ('transmit', 'receive', 'both')),
duration_sec INTEGER NOT NULL DEFAULT 10,
tx_avg_mbps DOUBLE PRECISION,
rx_avg_mbps DOUBLE PRECISION,
ping_rtt_ms INTEGER,
ping_loss_pct INTEGER,
ping_error TEXT,
status TEXT NOT NULL DEFAULT 'done' CHECK (status IN ('done', 'error')),
error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_uptime_speed_test_runs_created_at ON uptime_speed_test_runs(created_at);
CREATE TABLE IF NOT EXISTS certificate_issue_jobs (
id TEXT PRIMARY KEY,
status TEXT NOT NULL DEFAULT 'queued' CHECK (status IN ('queued', 'running', 'done', 'failed')),
step TEXT NOT NULL DEFAULT 'queued',
source TEXT NOT NULL DEFAULT 'manual' CHECK (source IN ('manual', 'scheduler')),
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
cert_name TEXT NOT NULL,
domain_names JSONB NOT NULL,
key_type TEXT NOT NULL DEFAULT 'rsa2048',
trust_store TEXT NOT NULL DEFAULT 'www,api',
requested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ,
error TEXT
);
CREATE TABLE IF NOT EXISTS backup_entries (
id TEXT PRIMARY KEY,
server_id BIGINT REFERENCES servers(id) ON DELETE SET NULL,
server_name TEXT NOT NULL,
filename TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
kind TEXT NOT NULL DEFAULT 'manual' CHECK (kind IN ('manual', 'auto')),
notes TEXT,
created_at TIMESTAMPTZ NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_backup_entries_filename ON backup_entries(filename);
CREATE INDEX IF NOT EXISTS idx_backup_entries_server_created ON backup_entries(server_id, created_at);
CREATE TABLE IF NOT EXISTS alert_groups (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
combine_mode TEXT NOT NULL DEFAULT 'any' CHECK (combine_mode IN ('any', 'all')),
enabled BOOLEAN NOT NULL DEFAULT TRUE,
cooldown_override TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS alert_rules (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
type TEXT NOT NULL,
target TEXT NOT NULL,
condition TEXT NOT NULL,
severity TEXT NOT NULL CHECK (severity IN ('critical', 'warning', 'info')),
enabled BOOLEAN NOT NULL DEFAULT TRUE,
cooldown TEXT NOT NULL DEFAULT '',
rule_chat_id TEXT NOT NULL DEFAULT '',
confirm_stability_sec INTEGER,
recovery_mode TEXT NOT NULL DEFAULT 'always' CHECK (recovery_mode IN ('always', 'never', 'conditional')),
recovery_stability_sec INTEGER,
group_id TEXT REFERENCES alert_groups(id) ON DELETE SET NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS alert_rule_targets (
id TEXT PRIMARY KEY,
rule_id TEXT NOT NULL REFERENCES alert_rules(id) ON DELETE CASCADE,
target TEXT NOT NULL,
sort_index INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS idx_alert_rule_targets_rule ON alert_rule_targets(rule_id);
CREATE TABLE IF NOT EXISTS alert_rule_conditions (
id TEXT PRIMARY KEY,
rule_id TEXT NOT NULL REFERENCES alert_rules(id) ON DELETE CASCADE,
condition_line TEXT NOT NULL,
sort_index INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS idx_alert_rule_conditions_rule ON alert_rule_conditions(rule_id);
CREATE TABLE IF NOT EXISTS alert_engine_state (
scope_key TEXT PRIMARY KEY,
last_fired_at TEXT NOT NULL DEFAULT '',
last_payload_hash TEXT
);
CREATE TABLE IF NOT EXISTS alert_engine_prev_live (
kind TEXT PRIMARY KEY CHECK (kind IN ('gre', 'bgp')),
payload_json JSONB NOT NULL DEFAULT '{}'::jsonb,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS alert_engine_confirm_pending (
rule_id TEXT PRIMARY KEY,
payload_hash TEXT NOT NULL,
since_at TIMESTAMPTZ NOT NULL
);
CREATE TABLE IF NOT EXISTS alert_history (
id TEXT PRIMARY KEY,
rule_id TEXT,
group_id TEXT,
rule_name TEXT NOT NULL,
severity TEXT NOT NULL CHECK (severity IN ('critical', 'warning', 'info')),
message TEXT NOT NULL,
sent_ok BOOLEAN NOT NULL DEFAULT TRUE,
fired_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_alert_history_fired_at ON alert_history(fired_at);
CREATE INDEX IF NOT EXISTS idx_alert_history_rule_id ON alert_history(rule_id);
CREATE TABLE IF NOT EXISTS alert_outbox (
id TEXT PRIMARY KEY,
dedupe_key TEXT NOT NULL,
channel TEXT NOT NULL DEFAULT 'telegram' CHECK (channel IN ('telegram')),
status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'sent', 'failed')),
retry_count INTEGER NOT NULL DEFAULT 0,
max_retries INTEGER NOT NULL DEFAULT 3,
next_attempt_at TIMESTAMPTZ NOT NULL,
payload_json JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
sent_at TIMESTAMPTZ,
last_error TEXT
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_alert_outbox_dedupe ON alert_outbox(dedupe_key);
CREATE INDEX IF NOT EXISTS idx_alert_outbox_status_next_attempt ON alert_outbox(status, next_attempt_at);
CREATE INDEX IF NOT EXISTS idx_alert_outbox_pending_next ON alert_outbox(next_attempt_at) WHERE status = 'pending';
CREATE TABLE IF NOT EXISTS scheduler_runs (
id TEXT PRIMARY KEY,
job_key TEXT NOT NULL,
started_at TIMESTAMPTZ NOT NULL,
finished_at TIMESTAMPTZ NOT NULL,
status TEXT NOT NULL CHECK (status IN ('ok', 'error')),
error TEXT,
duration_ms INTEGER NOT NULL DEFAULT 0,
result_json JSONB
);
CREATE INDEX IF NOT EXISTS idx_scheduler_runs_job_time ON scheduler_runs(job_key, started_at);
CREATE TABLE IF NOT EXISTS events (
id TEXT PRIMARY KEY,
created_at TIMESTAMPTZ NOT NULL,
level TEXT NOT NULL CHECK (level IN ('critical', 'warning', 'info')),
event_type TEXT NOT NULL,
source_module TEXT NOT NULL,
title TEXT NOT NULL,
message TEXT NOT NULL,
entity_type TEXT,
entity_id TEXT,
payload_json JSONB
);
CREATE INDEX IF NOT EXISTS idx_events_created_at ON events(created_at DESC);
CREATE INDEX IF NOT EXISTS idx_events_level_created_at ON events(level, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_events_source_created_at ON events(source_module, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_events_event_type_created_at ON events(event_type, created_at DESC);
CREATE TABLE IF NOT EXISTS app_users (
id TEXT PRIMARY KEY,
name TEXT NOT NULL DEFAULT '',
login TEXT NOT NULL UNIQUE,
email TEXT NOT NULL DEFAULT '',
role TEXT NOT NULL DEFAULT 'viewer' CHECK (role IN ('admin', 'operator', 'viewer')),
active BOOLEAN NOT NULL DEFAULT TRUE,
avatar TEXT NOT NULL DEFAULT '',
last_seen TIMESTAMPTZ,
sections_json JSONB NOT NULL DEFAULT '[]'::jsonb,
servers_json JSONB NOT NULL DEFAULT '[]'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS user_interface_bindings (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE,
server_id BIGINT NOT NULL REFERENCES servers(id) ON DELETE CASCADE,
interface_name TEXT NOT NULL,
interface_type TEXT NOT NULL DEFAULT 'other' CHECK (interface_type IN ('ether', 'gre', 'wg', 'other')),
peer_public_key TEXT NOT NULL DEFAULT '',
peer_name TEXT NOT NULL DEFAULT '',
comment TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (server_id, interface_name, peer_public_key)
);
CREATE INDEX IF NOT EXISTS idx_user_iface_bind_user ON user_interface_bindings(user_id);
CREATE TABLE IF NOT EXISTS internet_path_snapshots (
id BIGINT GENERATED BY DEFAULT AS IDENTITY,
sampled_at TIMESTAMPTZ NOT NULL,
payload_json JSONB NOT NULL,
PRIMARY KEY (id, sampled_at)
) PARTITION BY RANGE (sampled_at);
CREATE INDEX IF NOT EXISTS idx_internet_path_snapshots_sampled ON internet_path_snapshots(sampled_at);
CREATE INDEX IF NOT EXISTS idx_internet_path_snapshots_brin ON internet_path_snapshots USING BRIN (sampled_at);
INSERT INTO traffic_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO traffic_flow_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO uptime_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO evobgp_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO servers_api_ping_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO internet_path_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO alert_telegram_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO acme_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO certificate_renew_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO backup_schedule_settings (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO alert_engine_cursor (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
INSERT INTO data_migration (id) VALUES (1) ON CONFLICT (id) DO NOTHING;
+4 -4
View File
@@ -1,13 +1,13 @@
{ {
"version": "7", "version": "7",
"dialect": "sqlite", "dialect": "postgresql",
"entries": [ "entries": [
{ {
"idx": 0, "idx": 0,
"version": "6", "version": "7",
"when": 1777572014210, "when": 1777572014210,
"tag": "0000_living_xorn", "tag": "0000_postgresql",
"breakpoints": true "breakpoints": true
} }
] ]
} }
+8 -3
View File
@@ -1,7 +1,7 @@
{ {
"name": "mikrotik-manager-backend", "name": "mikrotik-manager-backend",
"version": "1.0.0", "version": "1.0.0",
"description": "MikroTik Manager backend — Fastify + Drizzle + SQLite", "description": "MikroTik Manager backend — Fastify + Drizzle + PostgreSQL",
"type": "module", "type": "module",
"scripts": { "scripts": {
"dev": "tsx watch src/index.ts", "dev": "tsx watch src/index.ts",
@@ -11,11 +11,14 @@
"db:generate": "drizzle-kit generate", "db:generate": "drizzle-kit generate",
"db:migrate": "drizzle-kit migrate", "db:migrate": "drizzle-kit migrate",
"db:studio": "drizzle-kit studio", "db:studio": "drizzle-kit studio",
"db:migrate-from-sqlite": "tsx src/scripts/migrate-sqlite-to-pg.ts",
"test:auth": "tsx src/lib/permissions.test.ts && tsx src/plugins/auth.smoke.test.ts", "test:auth": "tsx src/lib/permissions.test.ts && tsx src/plugins/auth.smoke.test.ts",
"test:wireguard": "npx tsx src/services/wireguard-config.test.ts", "test:wireguard": "npx tsx src/services/wireguard-config.test.ts",
"test:traffic-rate": "tsx src/services/traffic-rate.test.ts", "test:traffic-rate": "tsx src/services/traffic-rate.test.ts",
"test:traffic-flow": "tsx src/services/traffic-flow-parse.test.ts && tsx src/services/traffic-flow-map-exporter.test.ts && tsx src/services/traffic-flow-ifaces.test.ts && tsx src/services/traffic-flow-dedup.test.ts && tsx src/services/traffic-flow-planes.test.ts && tsx src/services/traffic-flow-ip.test.ts && tsx src/services/traffic-flow-classify.test.ts && tsx src/services/traffic-flow-ripe.test.ts && tsx src/services/traffic-flow-brands.test.ts && tsx src/services/traffic-flow-ingest.test.ts && tsx src/services/traffic-flow-analytics.test.ts && tsx src/services/traffic-flow-map-hops.test.ts && tsx src/services/traffic-flow-hardening.test.ts && tsx src/services/traffic-flow-purge.test.ts && tsx src/services/sqlite-write-opt.test.ts", "test:traffic-flow": "tsx src/services/traffic-flow-parse.test.ts && tsx src/services/traffic-flow-map-exporter.test.ts && tsx src/services/traffic-flow-ifaces.test.ts && tsx src/services/traffic-flow-dedup.test.ts && tsx src/services/traffic-flow-planes.test.ts && tsx src/services/traffic-flow-ip.test.ts && tsx src/services/traffic-flow-classify.test.ts && tsx src/services/traffic-flow-ripe.test.ts && tsx src/services/traffic-flow-brands.test.ts && tsx src/services/traffic-flow-ingest.test.ts && tsx src/services/traffic-flow-analytics.test.ts && tsx src/services/traffic-flow-map-hops.test.ts && tsx src/services/traffic-flow-purge.test.ts",
"test:users": "tsx src/modules/users/iface-type.test.ts && tsx src/modules/users/bindings.test.ts" "test:users": "tsx src/modules/users/iface-type.test.ts && tsx src/modules/users/bindings.test.ts",
"test:pg": "tsx src/db/sql-bind.test.ts && tsx src/db/pg-schema.test.ts",
"test": "npm run test:alert-engine && npm run test:auth && npm run test:wireguard && npm run test:traffic-rate && npm run test:traffic-flow && npm run test:users && npm run test:pg"
}, },
"dependencies": { "dependencies": {
"@fastify/cors": "^11.2.0", "@fastify/cors": "^11.2.0",
@@ -28,12 +31,14 @@
"drizzle-orm": "^0.45.2", "drizzle-orm": "^0.45.2",
"fastify": "^5.8.5", "fastify": "^5.8.5",
"fastify-plugin": "^5.1.0", "fastify-plugin": "^5.1.0",
"pg": "^8.23.0",
"undici": "^8.1.0", "undici": "^8.1.0",
"zod": "^4.4.1" "zod": "^4.4.1"
}, },
"devDependencies": { "devDependencies": {
"@types/better-sqlite3": "^7.6.13", "@types/better-sqlite3": "^7.6.13",
"@types/node": "^22.15.3", "@types/node": "^22.15.3",
"@types/pg": "^8.23.1",
"drizzle-kit": "^0.31.10", "drizzle-kit": "^0.31.10",
"jose": "^6.2.11", "jose": "^6.2.11",
"pino-pretty": "^13.1.3", "pino-pretty": "^13.1.3",
+39
View File
@@ -0,0 +1,39 @@
import Database from "better-sqlite3"
import fs from "node:fs"
const path = process.argv[2] ?? "./mikrotik.db"
if (!fs.existsSync(path)) {
console.error("NO_SQLITE", path)
process.exit(2)
}
const st = fs.statSync(path)
const db = new Database(path, { readonly: true, fileMustExist: true })
const tables = db
.prepare(`SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY 1`)
.all()
console.log("FILE_BYTES", st.size)
console.log("TABLES", tables.length)
for (const t of tables) {
const c = db.prepare(`SELECT COUNT(*) AS c FROM "${t.name}"`).get()
console.log(`COUNT\t${t.name}\t${c.c}`)
}
try {
const d = db.prepare(`SELECT COUNT(*) AS c FROM (
SELECT dedupe_key FROM alert_outbox GROUP BY dedupe_key HAVING COUNT(*) > 1
)`).get()
console.log("DUP_DEDUPE_KEYS", d.c)
} catch (e) {
console.log("DUP_DEDUPE_ERR", e instanceof Error ? e.message : e)
}
for (const [label, sql] of [
["ORPHAN_FLOW_MINUTE", `SELECT COUNT(*) c FROM flow_minute_stats s WHERE s.server_id NOT IN (SELECT id FROM servers)`],
["ORPHAN_FLOW_BUCKETS", `SELECT COUNT(*) c FROM flow_buckets s WHERE s.server_id NOT IN (SELECT id FROM servers)`],
["ORPHAN_BINDINGS", `SELECT COUNT(*) c FROM user_interface_bindings s WHERE s.server_id NOT IN (SELECT id FROM servers)`],
]) {
try {
console.log(label, db.prepare(sql).get().c)
} catch (e) {
console.log(label + "_ERR", e instanceof Error ? e.message : e)
}
}
db.close()
+6
View File
@@ -11,7 +11,9 @@ function boolEnv(v: string | undefined, fallback: boolean): boolean {
const isProd = process.env.NODE_ENV === "production" const isProd = process.env.NODE_ENV === "production"
const envSchema = z.object({ const envSchema = z.object({
DATABASE_URL: z.string().default("postgres://mmapp:[email protected]:5432/mmapp"),
DATABASE_PATH: z.string().default("./mikrotik.db"), DATABASE_PATH: z.string().default("./mikrotik.db"),
SQLITE_IMPORT_FULL_HISTORY: z.boolean().default(false),
PORT: z.coerce.number().int().positive().default(8000), PORT: z.coerce.number().int().positive().default(8000),
CORS_ORIGIN: z.string().default("http://localhost:3000"), CORS_ORIGIN: z.string().default("http://localhost:3000"),
AUTH_REQUIRED: z.boolean().default(false), AUTH_REQUIRED: z.boolean().default(false),
@@ -21,7 +23,9 @@ const envSchema = z.object({
}) })
const raw = { const raw = {
DATABASE_URL: process.env.DATABASE_URL,
DATABASE_PATH: process.env.DATABASE_PATH, DATABASE_PATH: process.env.DATABASE_PATH,
SQLITE_IMPORT_FULL_HISTORY: boolEnv(process.env.SQLITE_IMPORT_FULL_HISTORY, false),
PORT: process.env.PORT, PORT: process.env.PORT,
CORS_ORIGIN: process.env.CORS_ORIGIN, CORS_ORIGIN: process.env.CORS_ORIGIN,
AUTH_REQUIRED: boolEnv(process.env.AUTH_REQUIRED, false), AUTH_REQUIRED: boolEnv(process.env.AUTH_REQUIRED, false),
@@ -53,7 +57,9 @@ if (parsed.data.AUTH_REQUIRED && parsed.data.AUTH_JWT_SECRET.length < 8) {
} }
export const env = { export const env = {
DATABASE_URL: parsed.data.DATABASE_URL,
DATABASE_PATH: parsed.data.DATABASE_PATH, DATABASE_PATH: parsed.data.DATABASE_PATH,
sqliteImportFullHistory: parsed.data.SQLITE_IMPORT_FULL_HISTORY,
PORT: parsed.data.PORT, PORT: parsed.data.PORT,
CORS_ORIGIN: parsed.data.CORS_ORIGIN, CORS_ORIGIN: parsed.data.CORS_ORIGIN,
authRequired: parsed.data.AUTH_REQUIRED, authRequired: parsed.data.AUTH_REQUIRED,
+33
View File
@@ -0,0 +1,33 @@
import { pool } from "./index.js"
import { applySqlMigrations } from "./migrate.js"
import { dropExpiredPartitions, ensurePartitionsAround } from "./partitions.js"
import { importSqliteToPostgres, shouldImportSqlite } from "./sqlite-import.js"
import { env } from "../config.js"
const ETL_LOCK = 8723101
export async function initDatabase(): Promise<void> {
await pool.query("SELECT 1")
await applySqlMigrations(pool)
await ensurePartitionsAround(pool)
const client = await pool.connect()
try {
await client.query("SELECT pg_advisory_lock($1)", [ETL_LOCK])
if (await shouldImportSqlite(pool, env.DATABASE_PATH)) {
console.log(`SQLite → PostgreSQL: импорт ${env.DATABASE_PATH}`)
const report = await importSqliteToPostgres(pool, env.DATABASE_PATH, { strict: true })
console.log(
`SQLite → PostgreSQL: готово за ${report.durationMs}ms, таблиц ${Object.keys(report.tables).length}`,
)
}
} finally {
try {
await client.query("SELECT pg_advisory_unlock($1)", [ETL_LOCK])
} catch {
/* ignore */
}
client.release()
}
await dropExpiredPartitions(pool)
await ensurePartitionsAround(pool)
}
+51 -1045
View File
File diff suppressed because it is too large Load Diff
+43
View File
@@ -0,0 +1,43 @@
/** JSONB from node-pg may already be an object; SQLite leftovers may be strings. */
export function parseJsonObject(value: unknown): Record<string, unknown> {
if (value && typeof value === "object" && !Array.isArray(value)) {
return value as Record<string, unknown>
}
if (typeof value === "string" && value.trim()) {
try {
const parsed = JSON.parse(value) as unknown
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
return parsed as Record<string, unknown>
}
} catch {
/* ignore */
}
}
return {}
}
export function parseJsonArray(value: unknown): unknown[] {
if (Array.isArray(value)) return value
if (typeof value === "string" && value.trim()) {
try {
const parsed = JSON.parse(value) as unknown
if (Array.isArray(parsed)) return parsed
} catch {
/* ignore */
}
}
return []
}
export function toJsonb(value: unknown): unknown {
if (value == null) return null
if (typeof value === "string") {
try {
return JSON.parse(value) as unknown
} catch {
return value
}
}
return value
}
+43
View File
@@ -0,0 +1,43 @@
import { readFileSync } from "node:fs"
import { dirname, join } from "node:path"
import { fileURLToPath } from "node:url"
import type { Pool } from "pg"
const MIGRATION_ID = "0000_postgresql"
function migrationsDir(): string {
const here = dirname(fileURLToPath(import.meta.url))
const candidates = [
join(here, "..", "..", "drizzle"),
join(process.cwd(), "drizzle"),
join(process.cwd(), "backend", "drizzle"),
]
for (const dir of candidates) {
try {
readFileSync(join(dir, `${MIGRATION_ID}.sql`), "utf8")
return dir
} catch {
/* try next */
}
}
throw new Error("Не найден backend/drizzle/0000_postgresql.sql")
}
export async function applySqlMigrations(pool: Pool): Promise<void> {
await pool.query(`
CREATE TABLE IF NOT EXISTS schema_migrations (
id TEXT PRIMARY KEY,
applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
)
`)
const { rows } = await pool.query<{ id: string }>(
`SELECT id FROM schema_migrations WHERE id = $1`,
[MIGRATION_ID],
)
if (rows.length > 0) return
const sql = readFileSync(join(migrationsDir(), `${MIGRATION_ID}.sql`), "utf8")
await pool.query(sql)
await pool.query(`INSERT INTO schema_migrations (id) VALUES ($1) ON CONFLICT (id) DO NOTHING`, [
MIGRATION_ID,
])
}
+129
View File
@@ -0,0 +1,129 @@
import type { Pool } from "pg"
export type PartitionKind = "day" | "week" | "month"
export interface PartitionSpec {
parent: string
kind: PartitionKind
keepDays: number
}
export const PARTITION_SPECS: PartitionSpec[] = [
{ parent: "flow_buckets", kind: "day", keepDays: 4 },
{ parent: "flow_minute_stats", kind: "day", keepDays: 4 },
{ parent: "flow_minute_dims", kind: "day", keepDays: 4 },
{ parent: "flow_daily_dims", kind: "month", keepDays: 420 },
{ parent: "traffic_samples", kind: "week", keepDays: 21 },
{ parent: "servers_rest_ping_samples", kind: "week", keepDays: 35 },
{ parent: "uptime_probe_samples", kind: "week", keepDays: 21 },
{ parent: "uptime_resource_samples", kind: "week", keepDays: 21 },
{ parent: "server_snapshots", kind: "week", keepDays: 21 },
{ parent: "internet_path_snapshots", kind: "week", keepDays: 21 },
]
function utcDate(d: Date): Date {
return new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate()))
}
function addUtcDays(d: Date, n: number): Date {
const x = utcDate(d)
x.setUTCDate(x.getUTCDate() + n)
return x
}
function startOfWeekUtc(d: Date): Date {
const x = utcDate(d)
const day = x.getUTCDay() || 7
x.setUTCDate(x.getUTCDate() - (day - 1))
return x
}
function startOfMonthUtc(d: Date): Date {
return new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), 1))
}
function fmtDay(d: Date): string {
return d.toISOString().slice(0, 10)
}
function partName(parent: string, kind: PartitionKind, start: Date): string {
if (kind === "day") return `${parent}_d_${fmtDay(start).replaceAll("-", "")}`
if (kind === "week") return `${parent}_w_${fmtDay(start).replaceAll("-", "")}`
return `${parent}_m_${start.getUTCFullYear()}${String(start.getUTCMonth() + 1).padStart(2, "0")}`
}
function rangeFor(kind: PartitionKind, ts: Date): { start: Date; end: Date } {
if (kind === "day") {
const start = utcDate(ts)
return { start, end: addUtcDays(start, 1) }
}
if (kind === "week") {
const start = startOfWeekUtc(ts)
return { start, end: addUtcDays(start, 7) }
}
const start = startOfMonthUtc(ts)
const end = new Date(Date.UTC(start.getUTCFullYear(), start.getUTCMonth() + 1, 1))
return { start, end }
}
export async function ensurePartitionFor(
pool: Pool,
parent: string,
kind: PartitionKind,
ts: Date,
): Promise<string> {
const { start, end } = rangeFor(kind, ts)
const name = partName(parent, kind, start)
const from = fmtDay(start)
const to = fmtDay(end)
await pool.query(
`CREATE TABLE IF NOT EXISTS ${name} PARTITION OF ${parent} FOR VALUES FROM ('${from}') TO ('${to}')`,
)
return name
}
export async function ensurePartitionsAround(pool: Pool, around = new Date()): Promise<void> {
for (const spec of PARTITION_SPECS) {
const daysAhead = spec.kind === "month" ? 40 : spec.kind === "week" ? 21 : 8
const start = addUtcDays(around, -spec.keepDays)
const end = addUtcDays(around, daysAhead)
for (let t = new Date(start.getTime()); t < end; ) {
await ensurePartitionFor(pool, spec.parent, spec.kind, t)
if (spec.kind === "day") t = addUtcDays(t, 1)
else if (spec.kind === "week") t = addUtcDays(t, 7)
else t = new Date(Date.UTC(t.getUTCFullYear(), t.getUTCMonth() + 1, 1))
}
}
}
export async function dropExpiredPartitions(pool: Pool, around = new Date()): Promise<void> {
for (const spec of PARTITION_SPECS) {
const cutoff = addUtcDays(around, -spec.keepDays)
const { rows } = await pool.query<{ relname: string }>(
`SELECT c.relname
FROM pg_inherits i
JOIN pg_class c ON c.oid = i.inhrelid
JOIN pg_class p ON p.oid = i.inhparent
WHERE p.relname = $1`,
[spec.parent],
)
for (const row of rows) {
const m = row.relname.match(/_([dwm])_(\d{8}|\d{6})$/)
if (!m) continue
const stamp = m[2]
let start: Date
if (stamp.length === 6) {
start = new Date(Date.UTC(Number(stamp.slice(0, 4)), Number(stamp.slice(4, 6)) - 1, 1))
} else {
start = new Date(`${stamp.slice(0, 4)}-${stamp.slice(4, 6)}-${stamp.slice(6, 8)}T00:00:00Z`)
}
if (start < cutoff) {
await pool.query(`DROP TABLE IF EXISTS ${row.relname}`)
}
}
}
}
export function specForParent(parent: string): PartitionSpec | undefined {
return PARTITION_SPECS.find((s) => s.parent === parent)
}
+56
View File
@@ -0,0 +1,56 @@
import assert from "node:assert/strict"
import { withPgOrSkip } from "../test/pg.js"
import { dbQuery, pool } from "./index.js"
import { ensurePartitionFor } from "./partitions.js"
if (!(await withPgOrSkip())) {
console.log("pg-schema.test.ts: skip")
process.exit(0)
}
{
const { rows } = await dbQuery<{ n: string }>(`SELECT COUNT(*)::text AS n FROM servers`)
assert.ok(rows[0])
}
{
await dbQuery(`
INSERT INTO servers (name, host) VALUES ('pg-schema-test', '127.0.0.1')
`)
const { rows } = await dbQuery<{ id: number }>(`SELECT id FROM servers WHERE name = 'pg-schema-test' LIMIT 1`)
const id = rows[0]?.id
assert.ok(id)
await ensurePartitionFor(pool, "flow_buckets", "day", new Date())
const bucketAt = new Date().toISOString()
await dbQuery(`
INSERT INTO flow_buckets (
server_id, bucket_at, src, dst, proto, src_port, dst_port, bytes, packets, in_iface
) VALUES ($1, $2, '10.0.0.1', '8.8.8.8', 6, 1, 443, 10, 1, 'wg-flow')
ON CONFLICT DO NOTHING
`, [id, bucketAt])
await dbQuery(`DELETE FROM flow_buckets WHERE server_id = $1`, [id])
await dbQuery(`DELETE FROM servers WHERE id = $1`, [id])
}
{
await dbQuery(`
INSERT INTO alert_outbox (id, dedupe_key, payload_json, next_attempt_at)
VALUES ('pg-test-1', 'pg-dedupe-key', '{}'::jsonb, now())
ON CONFLICT (dedupe_key) DO NOTHING
`)
let threw = false
try {
await dbQuery(`
INSERT INTO alert_outbox (id, dedupe_key, payload_json, next_attempt_at)
VALUES ('pg-test-2', 'pg-dedupe-key', '{}'::jsonb, now())
`)
} catch {
threw = true
}
assert.equal(threw, true, "alert_outbox.dedupe_key UNIQUE")
await dbQuery(`DELETE FROM alert_outbox WHERE dedupe_key = 'pg-dedupe-key'`)
}
console.log("pg-schema.test.ts: ok")
+348 -364
View File
File diff suppressed because it is too large Load Diff
+21
View File
@@ -0,0 +1,21 @@
import assert from "node:assert/strict"
import { bindSql } from "./sql-bind.js"
{
const q = bindSql("SELECT 1")
assert.deepEqual(q, { text: "SELECT 1", values: [] })
}
{
const q = bindSql("SELECT * FROM t WHERE a = ? AND b = ?", [1, "x"])
assert.equal(q.text, "SELECT * FROM t WHERE a = $1 AND b = $2")
assert.deepEqual(q.values, [1, "x"])
}
{
const q = bindSql("INSERT INTO t (a, b) VALUES (@a, @b)", { a: 3, b: "y" })
assert.equal(q.text, "INSERT INTO t (a, b) VALUES ($1, $2)")
assert.deepEqual(q.values, [3, "y"])
}
console.log("sql-bind.test.ts: ok")
+22
View File
@@ -0,0 +1,22 @@
/** Convert SQLite-style @name / ? placeholders to node-pg $n. */
export function bindSql(
sql: string,
params?: unknown[] | Record<string, unknown>,
): { text: string; values: unknown[] } {
if (params == null) return { text: sql, values: [] }
if (Array.isArray(params)) {
let i = 0
const text = sql.replace(/\?/g, () => {
i += 1
return `$${i}`
})
return { text, values: params }
}
const values: unknown[] = []
const text = sql.replace(/@([a-zA-Z_][a-zA-Z0-9_]*)/g, (_, name: string) => {
values.push(params[name])
return `$${values.length}`
})
return { text, values }
}
+479
View File
@@ -0,0 +1,479 @@
import { createHash } from "node:crypto"
import { existsSync, readFileSync } from "node:fs"
import Database from "better-sqlite3"
import type { Pool } from "pg"
import { env } from "../config.js"
import { ensurePartitionFor, specForParent } from "./partitions.js"
export interface ImportReport {
sqlitePath: string
sqliteSha256: string
tables: Record<string, { sqlite: number; copied: number; skipped: number }>
rejects: string[]
durationMs: number
}
const SNAPSHOT_RETENTION_DAYS = 14
type ColKind = "ts" | "date" | "bool" | "json" | "json-null" | "bigint-id" | "int" | "text" | "num"
interface TableCopy {
table: string
columns: Array<[string, ColKind]>
timeCol?: string
retentionDays?: number
identity?: boolean
upsert?: boolean
}
const TABLES: TableCopy[] = [
{ table: "servers", identity: true, columns: [
["id", "int"], ["name", "text"], ["host", "text"], ["port", "int"], ["username", "text"],
["password", "text"], ["use_ssl", "bool"], ["verify_ssl", "bool"], ["type", "text"],
["site", "text"], ["country", "text"], ["asn", "text"], ["comment", "text"], ["enabled", "bool"],
["lan_subnet", "text"], ["wan_uplinks", "json"], ["mgmt_tunnel_ip", "text"],
["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "traffic_settings", upsert: true, columns: [
["id", "int"], ["enabled", "bool"], ["interval_sec", "int"], ["retention_days", "int"],
["last_collected_at", "ts"], ["last_duration_ms", "int"], ["last_error", "text"],
["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "servers_api_ping_settings", upsert: true, columns: [
["id", "int"], ["enabled", "bool"], ["interval_sec", "int"],
["last_collected_at", "ts"], ["last_duration_ms", "int"], ["last_error", "text"],
["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "traffic_flow_settings", upsert: true, columns: [
["id", "int"], ["enabled", "bool"], ["collector_ip", "text"], ["flow_listen_port", "int"],
["wg_listen_port", "int"], ["prefix", "text"], ["public_endpoint", "text"],
["host_public_key", "text"], ["host_private_key", "text"], ["hub_server_id", "int"],
["retention_hours", "int"], ["top_n", "int"], ["map_service_min_share_pct", "num"],
["last_datagram_at", "ts"], ["last_exporter_ip", "text"], ["last_error", "text"],
["packets_received", "int"], ["peers_json", "json"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "uptime_settings", upsert: true, columns: [
["id", "int"], ["enabled", "bool"], ["resources_enabled", "bool"], ["ping_enabled", "bool"],
["speed_enabled", "bool"], ["interval_sec", "int"], ["probe_interval_sec", "int"],
["speed_interval_sec", "int"], ["retention_days", "int"], ["last_collected_at", "ts"],
["last_duration_ms", "int"], ["last_error", "text"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "evobgp_settings", upsert: true, columns: [
["id", "int"], ["base_url", "text"], ["api_key", "text"], ["enabled", "bool"], ["updated_at", "ts"],
]},
{ table: "alert_telegram_settings", upsert: true, columns: [
["id", "int"], ["bot_token", "text"], ["chat_id", "text"], ["message_thread_id", "int"], ["updated_at", "ts"],
]},
{ table: "acme_settings", upsert: true, columns: [
["id", "int"], ["directory_url", "text"], ["cloudflare_api_token", "text"],
["default_zone_id", "text"], ["account_private_key", "text"], ["updated_at", "ts"],
]},
{ table: "certificate_renew_settings", upsert: true, columns: [
["id", "int"], ["enabled", "bool"], ["interval_sec", "int"], ["renew_before_days", "int"],
["last_collected_at", "ts"], ["last_duration_ms", "int"], ["last_error", "text"], ["updated_at", "ts"],
]},
{ table: "backup_schedule_settings", upsert: true, columns: [
["id", "int"], ["enabled", "bool"], ["frequency", "text"], ["hour", "int"], ["minute", "int"],
["week_day", "int"], ["month_day", "int"], ["keep_count", "int"], ["format", "text"],
["server_ids_json", "json"], ["last_run_at", "ts"], ["last_duration_ms", "int"],
["last_error", "text"], ["updated_at", "ts"],
]},
{ table: "internet_path_settings", upsert: true, columns: [
["id", "int"], ["enabled", "bool"], ["interval_sec", "int"], ["retention_days", "int"],
["last_collected_at", "ts"], ["last_duration_ms", "int"], ["last_error", "text"],
["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "alert_engine_cursor", upsert: true, columns: [
["id", "int"], ["last_source_finished_at", "ts"], ["updated_at", "ts"],
]},
{ table: "filter_rules", identity: true, columns: [
["id", "int"], ["server_id", "int"], ["sort_order", "int"], ["community", "text"],
["community_name", "text"], ["action", "text"], ["gateway", "text"],
["gateway_tunnel_id", "text"], ["description", "text"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "recursive_routes", identity: true, columns: [
["id", "int"], ["server_id", "int"], ["sort_order", "int"], ["dst_address", "text"],
["gateway", "text"], ["distance", "int"], ["scope", "int"], ["target_scope", "int"],
["routing_table", "text"], ["check_gateway", "text"], ["country", "text"], ["comment", "text"],
["disabled", "bool"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "server_snapshots", identity: true, timeCol: "polled_at", retentionDays: SNAPSHOT_RETENTION_DAYS, columns: [
["id", "int"], ["server_id", "int"], ["polled_at", "ts"], ["status", "text"], ["latency_ms", "num"],
["ros_version", "text"], ["board_name", "text"], ["uptime", "text"], ["cpu_load", "int"],
["free_memory", "int"], ["total_memory", "int"], ["identity_name", "text"],
["raw_interfaces", "json-null"], ["raw_ip_addresses", "json-null"],
]},
{ table: "traffic_samples", identity: true, timeCol: "sampled_at", retentionDays: 14, columns: [
["id", "int"], ["server_id", "int"], ["interface_name", "text"], ["peer_public_key", "text"],
["sampled_at", "ts"], ["rx_bytes", "int"], ["tx_bytes", "int"], ["rx_bps", "int"], ["tx_bps", "int"],
["running", "bool"], ["disabled", "bool"],
]},
{ table: "servers_rest_ping_samples", identity: true, timeCol: "sampled_at", retentionDays: 30, columns: [
["id", "int"], ["server_id", "int"], ["sampled_at", "ts"], ["ok", "bool"], ["latency_ms", "int"], ["error", "text"],
]},
{ table: "flow_buckets", timeCol: "bucket_at", retentionDays: 2, columns: [
["server_id", "int"], ["bucket_at", "ts"], ["src", "text"], ["dst", "text"], ["proto", "int"],
["src_port", "int"], ["dst_port", "int"], ["bytes", "int"], ["packets", "int"],
["in_iface", "text"], ["out_iface", "text"], ["next_hop", "text"],
["flow_start_ms", "int"], ["flow_end_ms", "int"],
]},
{ table: "flow_minute_stats", timeCol: "bucket_at", retentionDays: 3, columns: [
["server_id", "int"], ["bucket_at", "ts"], ["bytes", "int"], ["packets", "int"],
["unique_src", "int"], ["unique_dst", "int"], ["conversations", "int"],
]},
{ table: "flow_minute_dims", timeCol: "bucket_at", retentionDays: 3, columns: [
["server_id", "int"], ["bucket_at", "ts"], ["dim", "text"], ["key", "text"], ["bytes", "int"], ["packets", "int"],
]},
{ table: "flow_daily_dims", timeCol: "day", retentionDays: 396, columns: [
["server_id", "int"], ["day", "date"], ["dim", "text"], ["key", "text"], ["bytes", "int"], ["packets", "int"],
]},
{ table: "flow_ip_meta", columns: [
["prefix", "text"], ["asn", "int"], ["country", "text"], ["lat", "num"], ["lng", "num"],
["holder", "text"], ["ok", "int"], ["fetched_at", "ts"],
]},
{ table: "flow_asn_meta", columns: [
["asn", "int"], ["holder", "text"], ["fetched_at", "ts"],
]},
{ table: "uptime_probes", columns: [
["id", "text"], ["src_server_id", "int"], ["src_interface", "text"], ["name", "text"],
["target", "text"], ["probe_filter", "text"], ["enabled", "bool"], ["interval_sec", "int"],
["show_on_dashboard", "bool"], ["sort_order", "int"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "uptime_probe_samples", identity: true, timeCol: "sampled_at", retentionDays: 14, columns: [
["id", "int"], ["probe_id", "text"], ["sampled_at", "ts"], ["rtt_ms", "int"], ["loss_pct", "int"], ["status", "text"],
]},
{ table: "uptime_resource_samples", identity: true, timeCol: "sampled_at", retentionDays: 14, columns: [
["id", "int"], ["server_id", "int"], ["sampled_at", "ts"], ["status", "text"], ["cpu_load", "int"],
["free_memory", "int"], ["total_memory", "int"], ["free_hdd_space", "int"], ["total_hdd_space", "int"],
["uptime_seconds", "int"], ["board_name", "text"], ["ros_version", "text"],
]},
{ table: "uptime_speed_probes", columns: [
["id", "text"], ["src_server_id", "int"], ["dst_server_id", "int"], ["src_interface", "text"],
["dst_interface", "text"], ["protocol", "text"], ["direction", "text"], ["duration_sec", "int"],
["enabled", "bool"], ["last_run_at", "ts"], ["last_tx_avg_mbps", "num"], ["last_rx_avg_mbps", "num"],
["last_status", "text"], ["last_error", "text"], ["last_ping_rtt_ms", "int"],
["last_ping_loss_pct", "int"], ["last_ping_at", "ts"], ["last_ping_error", "text"],
["sort_order", "int"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "uptime_speed_test_runs", columns: [
["id", "text"], ["probe_id", "text"], ["src_server_id", "int"], ["dst_server_id", "int"],
["src_interface", "text"], ["dst_interface", "text"], ["src_address", "text"], ["dst_address", "text"],
["src_interface_address", "text"], ["dst_interface_address", "text"], ["protocol", "text"],
["direction", "text"], ["duration_sec", "int"], ["tx_avg_mbps", "num"], ["rx_avg_mbps", "num"],
["ping_rtt_ms", "int"], ["ping_loss_pct", "int"], ["ping_error", "text"], ["status", "text"],
["error", "text"], ["created_at", "ts"],
]},
{ table: "app_users", columns: [
["id", "text"], ["name", "text"], ["login", "text"], ["email", "text"], ["role", "text"],
["active", "bool"], ["avatar", "text"], ["last_seen", "ts"], ["sections_json", "json"],
["servers_json", "json"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "user_interface_bindings", columns: [
["id", "text"], ["user_id", "text"], ["server_id", "int"], ["interface_name", "text"],
["interface_type", "text"], ["peer_public_key", "text"], ["peer_name", "text"],
["comment", "text"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "alert_groups", columns: [
["id", "text"], ["name", "text"], ["combine_mode", "text"], ["enabled", "bool"],
["cooldown_override", "text"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "alert_rules", columns: [
["id", "text"], ["name", "text"], ["type", "text"], ["target", "text"], ["condition", "text"],
["severity", "text"], ["enabled", "bool"], ["cooldown", "text"], ["rule_chat_id", "text"],
["confirm_stability_sec", "int"], ["recovery_mode", "text"], ["recovery_stability_sec", "int"],
["group_id", "text"], ["created_at", "ts"], ["updated_at", "ts"],
]},
{ table: "alert_rule_targets", columns: [
["id", "text"], ["rule_id", "text"], ["target", "text"], ["sort_index", "int"],
]},
{ table: "alert_rule_conditions", columns: [
["id", "text"], ["rule_id", "text"], ["condition_line", "text"], ["sort_index", "int"],
]},
{ table: "alert_engine_state", columns: [
["scope_key", "text"], ["last_fired_at", "text"], ["last_payload_hash", "text"],
]},
{ table: "alert_engine_prev_live", columns: [
["kind", "text"], ["payload_json", "json"], ["updated_at", "ts"],
]},
{ table: "alert_engine_confirm_pending", columns: [
["rule_id", "text"], ["payload_hash", "text"], ["since_at", "ts"],
]},
{ table: "alert_history", columns: [
["id", "text"], ["rule_id", "text"], ["group_id", "text"], ["rule_name", "text"],
["severity", "text"], ["message", "text"], ["sent_ok", "bool"], ["fired_at", "ts"],
]},
{ table: "alert_outbox", columns: [
["id", "text"], ["dedupe_key", "text"], ["channel", "text"], ["status", "text"],
["retry_count", "int"], ["max_retries", "int"], ["next_attempt_at", "ts"],
["payload_json", "json"], ["created_at", "ts"], ["sent_at", "ts"], ["last_error", "text"],
]},
{ table: "certificate_issue_jobs", columns: [
["id", "text"], ["status", "text"], ["step", "text"], ["source", "text"],
["server_id", "bigint-id"], ["cert_name", "text"], ["domain_names", "json"],
["key_type", "text"], ["trust_store", "text"], ["requested_at", "ts"],
["started_at", "ts"], ["finished_at", "ts"], ["error", "text"],
]},
{ table: "backup_entries", columns: [
["id", "text"], ["server_id", "bigint-id"], ["server_name", "text"], ["filename", "text"],
["size_bytes", "int"], ["kind", "text"], ["notes", "text"], ["created_at", "ts"],
]},
{ table: "scheduler_runs", timeCol: "started_at", retentionDays: 30, columns: [
["id", "text"], ["job_key", "text"], ["started_at", "ts"], ["finished_at", "ts"],
["status", "text"], ["error", "text"], ["duration_ms", "int"], ["result_json", "json-null"],
]},
{ table: "events", timeCol: "created_at", retentionDays: 30, columns: [
["id", "text"], ["created_at", "ts"], ["level", "text"], ["event_type", "text"],
["source_module", "text"], ["title", "text"], ["message", "text"],
["entity_type", "text"], ["entity_id", "text"], ["payload_json", "json-null"],
]},
{ table: "internet_path_snapshots", identity: true, timeCol: "sampled_at", retentionDays: 14, columns: [
["id", "int"], ["sampled_at", "ts"], ["payload_json", "json"],
]},
]
function parseTs(value: unknown, strict: boolean, rejects: string[], ctx: string): string | null {
if (value == null || value === "") return null
const s = String(value)
try {
if (/^\d{4}-\d{2}-\d{2}T/.test(s)) return new Date(s).toISOString()
if (/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/.test(s)) {
return new Date(`${s.replace(" ", "T")}Z`).toISOString()
}
const d = new Date(s)
if (!Number.isNaN(d.getTime())) return d.toISOString()
} catch {
/* fallthrough */
}
const msg = `${ctx}: неразобранный timestamp ${s}`
if (strict) throw new Error(msg)
rejects.push(msg)
return null
}
function parseDate(value: unknown): string | null {
if (value == null || value === "") return null
return String(value).slice(0, 10)
}
function parseBool(value: unknown): boolean {
return value === true || value === 1 || value === "1" || value === "true"
}
function parseJson(value: unknown, fallback: unknown): unknown {
if (value == null || value === "") return fallback
if (typeof value !== "string") return value
try {
return JSON.parse(value) as unknown
} catch {
return fallback
}
}
function coerce(kind: ColKind, value: unknown, strict: boolean, rejects: string[], ctx: string): unknown {
switch (kind) {
case "ts":
return parseTs(value, strict, rejects, ctx)
case "date":
return parseDate(value)
case "bool":
return parseBool(value)
case "json":
return parseJson(value, [])
case "json-null":
return value == null || value === "" ? null : parseJson(value, null)
case "bigint-id": {
const t = String(value ?? "").trim()
if (!t) return null
const n = Number(t)
if (!Number.isFinite(n)) {
const msg = `${ctx}: server_id ${t}`
if (strict) throw new Error(msg)
rejects.push(msg)
return null
}
return n
}
case "int":
if (value == null || value === "") return null
return Number(value)
case "num":
if (value == null || value === "") return null
return Number(value)
case "text":
return value == null ? "" : String(value)
default:
return value == null ? null : String(value)
}
}
function sqliteTableExists(sqlite: Database.Database, name: string): boolean {
const row = sqlite.prepare(
`SELECT 1 AS ok FROM sqlite_master WHERE type = 'table' AND name = ?`,
).get(name) as { ok?: number } | undefined
return Boolean(row)
}
async function setval(pool: Pool, table: string): Promise<void> {
await pool.query(
`SELECT setval(
pg_get_serial_sequence($1, 'id'),
GREATEST(COALESCE((SELECT MAX(id) FROM ${table}), 1), 1),
true
)`,
[table],
)
}
async function copyTable(
sqlite: Database.Database,
pool: Pool,
spec: TableCopy,
opts: { strict: boolean; fullHistory: boolean; rejects: string[] },
): Promise<{ sqlite: number; copied: number; skipped: number }> {
if (!sqliteTableExists(sqlite, spec.table)) {
return { sqlite: 0, copied: 0, skipped: 0 }
}
let where = ""
if (spec.timeCol && spec.retentionDays && !opts.fullHistory) {
const cutoff = new Date(Date.now() - spec.retentionDays * 86400_000).toISOString()
where = ` WHERE ${spec.timeCol} >= '${cutoff.replace("T", " ").slice(0, 19)}' OR ${spec.timeCol} >= '${cutoff}'`
}
const total = (sqlite.prepare(`SELECT COUNT(*) AS c FROM ${spec.table}${where}`).get() as { c: number }).c
const part = specForParent(spec.table)
const cols = spec.columns.map(([c]) => c)
const placeholders = cols.map((_, i) => `$${i + 1}`).join(", ")
const conflictSql = spec.upsert
? `ON CONFLICT (id) DO UPDATE SET ${cols.filter((c) => c !== "id").map((c) => `${c} = EXCLUDED.${c}`).join(", ")}`
: `ON CONFLICT DO NOTHING`
const insertSql = `INSERT INTO ${spec.table} (${cols.join(", ")}) VALUES (${placeholders}) ${conflictSql}`
let copied = 0
let skipped = 0
const stmt = sqlite.prepare(`SELECT * FROM ${spec.table}${where}`)
const batch: unknown[][] = []
const flush = async () => {
if (batch.length === 0) return
const client = await pool.connect()
try {
await client.query("BEGIN")
for (const values of batch) {
await client.query(insertSql, values)
copied += 1
}
await client.query("COMMIT")
} catch (err) {
await client.query("ROLLBACK")
throw err
} finally {
client.release()
batch.length = 0
}
}
for (const row of stmt.iterate() as Iterable<Record<string, unknown>>) {
try {
if (part && spec.timeCol) {
const raw = row[spec.timeCol]
const ts = spec.columns.find((c) => c[0] === spec.timeCol)?.[1] === "date"
? `${String(raw).slice(0, 10)}T00:00:00Z`
: parseTs(raw, false, opts.rejects, spec.table)
if (ts) await ensurePartitionFor(pool, spec.table, part.kind, new Date(ts))
}
const values = spec.columns.map(([col, kind]) =>
coerce(kind, row[col], opts.strict, opts.rejects, `${spec.table}.${col}`),
)
if (spec.table === "certificate_issue_jobs" && values[4] == null) {
skipped += 1
continue
}
batch.push(values)
if (batch.length >= 200) await flush()
} catch (err) {
skipped += 1
const msg = `${spec.table}: ${err instanceof Error ? err.message : String(err)}`
opts.rejects.push(msg)
if (opts.strict) throw err
}
}
await flush()
if (spec.identity) {
try {
await setval(pool, spec.table)
} catch {
/* partitioned identity sequence name may differ */
}
}
return { sqlite: total, copied, skipped }
}
export function sqliteFileLooksPresent(path: string): boolean {
if (!existsSync(path) || path === ":memory:") return false
try {
const buf = readFileSync(path)
return buf.subarray(0, 16).toString("utf8").startsWith("SQLite format 3")
} catch {
return false
}
}
export async function shouldImportSqlite(pool: Pool, sqlitePath: string): Promise<boolean> {
if (!sqliteFileLooksPresent(sqlitePath)) return false
const marker = await pool.query<{ sqlite_imported_at: string | null }>(
`SELECT sqlite_imported_at FROM data_migration WHERE id = 1`,
)
if (marker.rows[0]?.sqlite_imported_at) return false
const servers = await pool.query<{ c: string }>(`SELECT COUNT(*)::text AS c FROM servers`)
if (Number(servers.rows[0]?.c ?? 0) > 0) return false
return true
}
export async function importSqliteToPostgres(
pool: Pool,
sqlitePath: string,
opts?: { dryRun?: boolean; strict?: boolean; fullHistory?: boolean },
): Promise<ImportReport> {
const started = Date.now()
const rejects: string[] = []
const sqlite = new Database(sqlitePath, { readonly: true, fileMustExist: true })
try {
sqlite.pragma("journal_mode = WAL")
} catch {
/* readonly */
}
const sha = createHash("sha256").update(readFileSync(sqlitePath)).digest("hex")
const report: ImportReport = {
sqlitePath,
sqliteSha256: sha,
tables: {},
rejects,
durationMs: 0,
}
const fullHistory = opts?.fullHistory ?? env.sqliteImportFullHistory
const strict = opts?.strict ?? true
if (opts?.dryRun) {
for (const spec of TABLES) {
if (!sqliteTableExists(sqlite, spec.table)) {
report.tables[spec.table] = { sqlite: 0, copied: 0, skipped: 0 }
continue
}
const c = (sqlite.prepare(`SELECT COUNT(*) AS c FROM ${spec.table}`).get() as { c: number }).c
report.tables[spec.table] = { sqlite: c, copied: 0, skipped: 0 }
}
sqlite.close()
report.durationMs = Date.now() - started
return report
}
for (const spec of TABLES) {
report.tables[spec.table] = await copyTable(sqlite, pool, spec, { strict, fullHistory, rejects })
}
sqlite.close()
await pool.query(
`UPDATE data_migration
SET sqlite_imported_at = now(), sqlite_path = $1, sqlite_sha256 = $2, report_json = $3::jsonb
WHERE id = 1`,
[sqlitePath, sha, JSON.stringify(report.tables)],
)
report.durationMs = Date.now() - started
if (strict && rejects.length > 0) {
throw new Error(`SQLite import strict: ${rejects.length} rejects\n${rejects.slice(0, 20).join("\n")}`)
}
return report
}
+6 -2
View File
@@ -3,6 +3,8 @@ import cors from "@fastify/cors"
import { serializerCompiler, validatorCompiler } from "@fastify/type-provider-zod" import { serializerCompiler, validatorCompiler } from "@fastify/type-provider-zod"
import { monitorEventLoopDelay } from "node:perf_hooks" import { monitorEventLoopDelay } from "node:perf_hooks"
import { env } from "./config.js" import { env } from "./config.js"
import { initDatabase } from "./db/bootstrap.js"
import { closePool } from "./db/index.js"
import authPlugin, { requireAuth } from "./plugins/auth.js" import authPlugin, { requireAuth } from "./plugins/auth.js"
import serversRoutes from "./routes/servers.js" import serversRoutes from "./routes/servers.js"
import bgpRoutes from "./routes/bgp.js" import bgpRoutes from "./routes/bgp.js"
@@ -132,8 +134,8 @@ export async function buildApp(opts?: {
await app.register(usersRoutes, { prefix: "/api" }) await app.register(usersRoutes, { prefix: "/api" })
if (opts?.startScheduler !== false) { if (opts?.startScheduler !== false) {
refreshScheduler() await refreshScheduler()
startTrafficFlowListener() await startTrafficFlowListener()
app.addHook("onClose", async () => { app.addHook("onClose", async () => {
stopScheduler() stopScheduler()
stopTrafficFlowListener() stopTrafficFlowListener()
@@ -149,6 +151,7 @@ const isMain =
if (isMain) { if (isMain) {
try { try {
await initDatabase()
const app = await buildApp() const app = await buildApp()
let shuttingDown = false let shuttingDown = false
const shutdown = async (code: number) => { const shutdown = async (code: number) => {
@@ -157,6 +160,7 @@ if (isMain) {
try { try {
stopTrafficFlowListener() stopTrafficFlowListener()
await app.close() await app.close()
await closePool()
} catch (err) { } catch (err) {
console.error(err) console.error(err)
} finally { } finally {
@@ -24,18 +24,7 @@ export type ListEventsParams = {
to?: string to?: string
} }
function parsePayload(payloadJson: string | null): Record<string, unknown> { import { parseJsonObject } from "../../../db/json.js"
if (!payloadJson) return {}
try {
const parsed = JSON.parse(payloadJson) as unknown
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
return parsed as Record<string, unknown>
}
} catch {
// keep backwards compatibility with malformed legacy payloads
}
return {}
}
function mapEventRow(row: typeof events.$inferSelect): EventItem { function mapEventRow(row: typeof events.$inferSelect): EventItem {
return { return {
@@ -48,13 +37,13 @@ function mapEventRow(row: typeof events.$inferSelect): EventItem {
message: row.message, message: row.message,
entityType: row.entityType ?? null, entityType: row.entityType ?? null,
entityId: row.entityId ?? null, entityId: row.entityId ?? null,
payload: parsePayload(row.payloadJson ?? null), payload: parseJsonObject(row.payloadJson),
} }
} }
export function insertEventsBatch(items: EventInsertInput[]) { export async function insertEventsBatch(items: EventInsertInput[]) {
if (items.length === 0) return if (items.length === 0) return
db.insert(events) await db.insert(events)
.values( .values(
items.map((item) => ({ items.map((item) => ({
id: item.id, id: item.id,
@@ -66,25 +55,23 @@ export function insertEventsBatch(items: EventInsertInput[]) {
message: item.message, message: item.message,
entityType: item.entityType ?? null, entityType: item.entityType ?? null,
entityId: item.entityId ?? null, entityId: item.entityId ?? null,
payloadJson: item.payload ? JSON.stringify(item.payload) : null, payloadJson: item.payload ?? null,
})), })),
) )
.run()
} }
export function listEvents(params: ListEventsParams): EventItem[] { export async function listEvents(params: ListEventsParams): Promise<EventItem[]> {
const where = and( const where = and(
params.level ? eq(events.level, params.level) : undefined, params.level ? eq(events.level, params.level) : undefined,
params.sourceModule ? eq(events.sourceModule, params.sourceModule) : undefined, params.sourceModule ? eq(events.sourceModule, params.sourceModule) : undefined,
params.from ? gte(events.createdAt, params.from) : undefined, params.from ? gte(events.createdAt, params.from) : undefined,
params.to ? lte(events.createdAt, params.to) : undefined, params.to ? lte(events.createdAt, params.to) : undefined,
) )
const rows = db const rows = await db
.select() .select()
.from(events) .from(events)
.where(where) .where(where)
.orderBy(desc(events.createdAt)) .orderBy(desc(events.createdAt))
.limit(params.limit) .limit(params.limit)
.all()
return rows.map(mapEventRow) return rows.map(mapEventRow)
} }
@@ -23,17 +23,17 @@ function normalizeEventInput(input: AppendEventBody): EventInsertInput {
} }
} }
export function appendEvent(input: AppendEventBody) { export async function appendEvent(input: AppendEventBody) {
const parsed = appendEventSchema.parse(input) const parsed = appendEventSchema.parse(input)
insertEventsBatch([normalizeEventInput(parsed)]) await insertEventsBatch([normalizeEventInput(parsed)])
} }
export function appendEvents(inputs: AppendEventBody[]) { export async function appendEvents(inputs: AppendEventBody[]) {
const rows = inputs.map((entry) => normalizeEventInput(appendEventSchema.parse(entry))) const rows = inputs.map((entry) => normalizeEventInput(appendEventSchema.parse(entry)))
insertEventsBatch(rows) await insertEventsBatch(rows)
} }
export function readEvents(query: Partial<ListEventsQuery>): EventItem[] { export async function readEvents(query: Partial<ListEventsQuery>): Promise<EventItem[]> {
const parsed = listEventsQuerySchema.parse(query) const parsed = listEventsQuerySchema.parse(query)
return listEvents(parsed) return await listEvents(parsed)
} }
@@ -1,29 +1,24 @@
import { parseJsonArray } from "../../../db/json.js"
import type { ServerRead, WanUplinkRead } from "../../../types/server.js" import type { ServerRead, WanUplinkRead } from "../../../types/server.js"
import type { ServerRow, SnapshotRow } from "../repository/servers-repository.js" import type { ServerRow, SnapshotRow } from "../repository/servers-repository.js"
function parseWanUplinksJson(raw: string | null | undefined): WanUplinkRead[] { function parseWanUplinksJson(raw: unknown): WanUplinkRead[] {
if (raw == null || raw === "") return [] const data = parseJsonArray(raw)
try { const out: WanUplinkRead[] = []
const data = JSON.parse(raw) as unknown for (const row of data) {
if (!Array.isArray(data)) return [] if (typeof row !== "object" || row === null) continue
const out: WanUplinkRead[] = [] const r = row as Record<string, unknown>
for (const row of data) { out.push({
if (typeof row !== "object" || row === null) continue id: typeof r.id === "string" ? r.id : "",
const r = row as Record<string, unknown> name: typeof r.name === "string" ? r.name : "",
out.push({ isp: typeof r.isp === "string" ? r.isp : "",
id: typeof r.id === "string" ? r.id : "", iface: typeof r.iface === "string" ? r.iface : "",
name: typeof r.name === "string" ? r.name : "", ip: typeof r.ip === "string" ? r.ip : "",
isp: typeof r.isp === "string" ? r.isp : "", maxDl: typeof r.maxDl === "number" && Number.isFinite(r.maxDl) ? r.maxDl : 0,
iface: typeof r.iface === "string" ? r.iface : "", maxUl: typeof r.maxUl === "number" && Number.isFinite(r.maxUl) ? r.maxUl : 0,
ip: typeof r.ip === "string" ? r.ip : "", })
maxDl: typeof r.maxDl === "number" && Number.isFinite(r.maxDl) ? r.maxDl : 0,
maxUl: typeof r.maxUl === "number" && Number.isFinite(r.maxUl) ? r.maxUl : 0,
})
}
return out
} catch {
return []
} }
return out
} }
export function toServerRead(server: ServerRow, snap: SnapshotRow | undefined): ServerRead { export function toServerRead(server: ServerRow, snap: SnapshotRow | undefined): ServerRead {
@@ -6,52 +6,52 @@ import { invalidateFlowCatalogCache } from "../../../services/traffic-flow-topol
export type ServerRow = typeof servers.$inferSelect export type ServerRow = typeof servers.$inferSelect
export type SnapshotRow = typeof serverSnapshots.$inferSelect export type SnapshotRow = typeof serverSnapshots.$inferSelect
export function listServerRows(): ServerRow[] { export async function listServerRows(): Promise<ServerRow[]> {
return db.select().from(servers).all() return await db.select().from(servers)
} }
export function getServerRowById(id: number): ServerRow | undefined { export async function getServerRowById(id: number): Promise<ServerRow | undefined> {
return db.select().from(servers).where(eq(servers.id, id)).limit(1).all()[0] const rows = await db.select().from(servers).where(eq(servers.id, id)).limit(1)
return rows[0]
} }
export function createServerRow( export async function createServerRow(
values: Omit<typeof servers.$inferInsert, "id">, values: Omit<typeof servers.$inferInsert, "id">,
): ServerRow { ): Promise<ServerRow> {
const [inserted] = db.insert(servers).values(values).returning().all() const [inserted] = await db.insert(servers).values(values).returning()
invalidateFlowCatalogCache() invalidateFlowCatalogCache()
return inserted return inserted
} }
export function updateServerRowById( export async function updateServerRowById(
id: number, id: number,
values: Partial<ServerRow>, values: Partial<ServerRow>,
): ServerRow { ): Promise<ServerRow> {
const [updated] = db.update(servers).set(values).where(eq(servers.id, id)).returning().all() const [updated] = await db.update(servers).set(values).where(eq(servers.id, id)).returning()
invalidateFlowCatalogCache() invalidateFlowCatalogCache()
return updated return updated
} }
export function deleteServerRowById(id: number): void { export async function deleteServerRowById(id: number): Promise<void> {
db.delete(servers).where(eq(servers.id, id)).run() await db.delete(servers).where(eq(servers.id, id))
invalidateFlowCatalogCache() invalidateFlowCatalogCache()
} }
export function listSnapshotsByServerId(serverId: number, limit: number): SnapshotRow[] { export async function listSnapshotsByServerId(serverId: number, limit: number): Promise<SnapshotRow[]> {
return db return db
.select() .select()
.from(serverSnapshots) .from(serverSnapshots)
.where(eq(serverSnapshots.serverId, serverId)) .where(eq(serverSnapshots.serverId, serverId))
.orderBy(desc(serverSnapshots.polledAt)) .orderBy(desc(serverSnapshots.polledAt))
.limit(limit) .limit(limit)
.all()
} }
export function getLatestSnapshot(serverId: number): SnapshotRow | undefined { export async function getLatestSnapshot(serverId: number): Promise<SnapshotRow | undefined> {
return db const rows = await db
.select() .select()
.from(serverSnapshots) .from(serverSnapshots)
.where(eq(serverSnapshots.serverId, serverId)) .where(eq(serverSnapshots.serverId, serverId))
.orderBy(desc(serverSnapshots.polledAt)) .orderBy(desc(serverSnapshots.polledAt))
.limit(1) .limit(1)
.all()[0] return rows[0]
} }
@@ -11,45 +11,46 @@ import {
import { toServerRead } from "../mapper/servers-mapper.js" import { toServerRead } from "../mapper/servers-mapper.js"
import type { ServerCreate, ServerRead, ServerUpdate, SnapshotRead } from "../../../types/server.js" import type { ServerCreate, ServerRead, ServerUpdate, SnapshotRead } from "../../../types/server.js"
export function listServersRead(): ServerRead[] { export async function listServersRead(): Promise<ServerRead[]> {
return listServerRows().map((server) => toServerRead(server, getLatestSnapshot(server.id))) const rows = await listServerRows()
return Promise.all(rows.map(async (server) => toServerRead(server, await getLatestSnapshot(server.id))))
} }
export function getServerReadById(id: number): ServerRead | undefined { export async function getServerReadById(id: number): Promise<ServerRead | undefined> {
const row = getServerRowById(id) const row = await getServerRowById(id)
if (!row) return undefined if (!row) return undefined
return toServerRead(row, getLatestSnapshot(row.id)) return toServerRead(row, await getLatestSnapshot(row.id))
} }
export function createServer(input: ServerCreate): ServerRead { export async function createServer(input: ServerCreate): Promise<ServerRead> {
const now = new Date().toISOString() const now = new Date().toISOString()
const { wanUplinks, ...rest } = input const { wanUplinks, ...rest } = input
const inserted = createServerRow({ const inserted = await createServerRow({
...rest, ...rest,
wanUplinks: JSON.stringify(wanUplinks ?? []), wanUplinks: wanUplinks ?? [],
createdAt: now, createdAt: now,
updatedAt: now, updatedAt: now,
}) })
return toServerRead(inserted, undefined) return toServerRead(inserted, undefined)
} }
export function updateServer(id: number, input: ServerUpdate): ServerRead { export async function updateServer(id: number, input: ServerUpdate): Promise<ServerRead> {
const { wanUplinks, ...rest } = input const { wanUplinks, ...rest } = input
const setPayload: Record<string, unknown> = { updatedAt: new Date().toISOString() } const setPayload: Record<string, unknown> = { updatedAt: new Date().toISOString() }
for (const [k, v] of Object.entries(rest)) { for (const [k, v] of Object.entries(rest)) {
if (v !== undefined) setPayload[k] = v if (v !== undefined) setPayload[k] = v
} }
if (wanUplinks !== undefined) { if (wanUplinks !== undefined) {
setPayload.wanUplinks = JSON.stringify(wanUplinks) setPayload.wanUplinks = wanUplinks
} }
const updated = updateServerRowById(id, setPayload as never) const updated = await updateServerRowById(id, setPayload as never)
return toServerRead(updated, getLatestSnapshot(updated.id)) return toServerRead(updated, await getLatestSnapshot(updated.id))
} }
export function deleteServer(id: number): void { export async function deleteServer(id: number): Promise<void> {
deleteServerRowById(id) await deleteServerRowById(id)
} }
export function listServerSnapshots(id: number, limit: number): SnapshotRead[] { export async function listServerSnapshots(id: number, limit: number): Promise<SnapshotRead[]> {
return listSnapshotsByServerId(id, limit).map(toSnapshotRead) return (await listSnapshotsByServerId(id, limit)).map(toSnapshotRead)
} }
+61 -77
View File
@@ -1,79 +1,7 @@
import assert from "node:assert/strict" import assert from "node:assert/strict"
import Database from "better-sqlite3"
import { normalizeBindingPeer, PeerBindError } from "./peer-bind.js" import { normalizeBindingPeer, PeerBindError } from "./peer-bind.js"
import { withPgOrSkip } from "../../test/pg.js"
const sqlite = new Database(":memory:") import { dbQuery } from "../../db/index.js"
sqlite.pragma("foreign_keys = ON")
sqlite.exec(`
CREATE TABLE servers (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL DEFAULT '',
host TEXT NOT NULL DEFAULT '127.0.0.1'
);
CREATE TABLE app_users (
id TEXT PRIMARY KEY,
name TEXT NOT NULL DEFAULT '',
login TEXT NOT NULL UNIQUE,
email TEXT NOT NULL DEFAULT '',
role TEXT NOT NULL DEFAULT 'viewer',
active INTEGER NOT NULL DEFAULT 1,
avatar TEXT NOT NULL DEFAULT '',
last_seen TEXT,
sections_json TEXT NOT NULL DEFAULT '[]',
servers_json TEXT NOT NULL DEFAULT '[]',
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE user_interface_bindings (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
server_id INTEGER NOT NULL,
interface_name TEXT NOT NULL,
interface_type TEXT NOT NULL DEFAULT 'other',
peer_public_key TEXT NOT NULL DEFAULT '',
peer_name TEXT NOT NULL DEFAULT '',
comment TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
FOREIGN KEY (user_id) REFERENCES app_users(id) ON DELETE CASCADE,
FOREIGN KEY (server_id) REFERENCES servers(id) ON DELETE CASCADE,
UNIQUE (server_id, interface_name, peer_public_key)
);
`)
sqlite.prepare("INSERT INTO servers (id, name, host) VALUES (1, 'jh', '10.0.0.1')").run()
sqlite.prepare("INSERT INTO app_users (id, name, login) VALUES ('u1', 'A', 'a.user')").run()
sqlite.prepare("INSERT INTO app_users (id, name, login) VALUES ('u2', 'B', 'b.user')").run()
sqlite.prepare(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type)
VALUES ('b1', 'u1', 1, 'gre-office', 'gre')
`).run()
assert.throws(
() => sqlite.prepare(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type)
VALUES ('b2', 'u2', 1, 'gre-office', 'gre')
`).run(),
/UNIQUE/i,
"один интерфейс на сервере — один пользователь",
)
sqlite.prepare(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type, peer_public_key, peer_name)
VALUES ('wg1', 'u1', 1, 'wg-server', 'wg', 'peer-key-aaa', 'phone')
`).run()
sqlite.prepare(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type, peer_public_key, peer_name)
VALUES ('wg2', 'u2', 1, 'wg-server', 'wg', 'peer-key-bbb', 'laptop')
`).run()
assert.throws(
() => sqlite.prepare(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type, peer_public_key)
VALUES ('wg3', 'u2', 1, 'wg-server', 'wg', 'peer-key-aaa')
`).run(),
/UNIQUE/i,
"один пир — один пользователь",
)
assert.throws( assert.throws(
() => normalizeBindingPeer("wg", ""), () => normalizeBindingPeer("wg", ""),
@@ -83,8 +11,64 @@ assert.throws(
assert.equal(normalizeBindingPeer("ether", "ignored"), "") assert.equal(normalizeBindingPeer("ether", "ignored"), "")
assert.equal(normalizeBindingPeer("wg", " abc "), "abc") assert.equal(normalizeBindingPeer("wg", " abc "), "abc")
sqlite.prepare("DELETE FROM app_users WHERE id = 'u1'").run() if (!(await withPgOrSkip())) {
const leftover = sqlite.prepare("SELECT COUNT(*) AS n FROM user_interface_bindings").get() as { n: number } console.log("users bindings unique+cascade tests skip")
assert.equal(leftover.n, 1, "каскад: привязки u1 удаляются, пир u2 остаётся") process.exit(0)
}
await dbQuery(`
INSERT INTO servers (id, name, host) VALUES (91001, 'jh-bind-test', '10.0.0.1')
ON CONFLICT (id) DO NOTHING
`)
await dbQuery(`
INSERT INTO app_users (id, name, login)
VALUES ('u-bind-1', 'A', 'a.bind.test'), ('u-bind-2', 'B', 'b.bind.test')
ON CONFLICT (id) DO NOTHING
`)
await dbQuery(`DELETE FROM user_interface_bindings WHERE server_id = 91001`)
await dbQuery(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type)
VALUES ('b-bind-1', 'u-bind-1', 91001, 'gre-office', 'gre')
`)
let uniqueIface = false
try {
await dbQuery(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type)
VALUES ('b-bind-2', 'u-bind-2', 91001, 'gre-office', 'gre')
`)
} catch {
uniqueIface = true
}
assert.equal(uniqueIface, true, "один интерфейс на сервере — один пользователь")
await dbQuery(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type, peer_public_key, peer_name)
VALUES ('wg-bind-1', 'u-bind-1', 91001, 'wg-server', 'wg', 'peer-key-aaa', 'phone')
`)
await dbQuery(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type, peer_public_key, peer_name)
VALUES ('wg-bind-2', 'u-bind-2', 91001, 'wg-server', 'wg', 'peer-key-bbb', 'laptop')
`)
let uniquePeer = false
try {
await dbQuery(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type, peer_public_key)
VALUES ('wg-bind-3', 'u-bind-2', 91001, 'wg-server', 'wg', 'peer-key-aaa')
`)
} catch {
uniquePeer = true
}
assert.equal(uniquePeer, true, "один пир — один пользователь")
await dbQuery(`DELETE FROM app_users WHERE id = 'u-bind-1'`)
const leftover = await dbQuery<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM user_interface_bindings WHERE server_id = 91001`,
)
assert.equal(leftover.rows[0]?.n, 1, "каскад: привязки u1 удаляются, пир u2 остаётся")
await dbQuery(`DELETE FROM app_users WHERE id IN ('u-bind-1', 'u-bind-2')`)
await dbQuery(`DELETE FROM servers WHERE id = 91001`)
console.log("users bindings unique+cascade tests ok") console.log("users bindings unique+cascade tests ok")
+4 -4
View File
@@ -27,10 +27,10 @@ function asBool(raw: unknown): boolean {
return s === "true" || s === "yes" || s === "1" return s === "true" || s === "yes" || s === "1"
} }
export function parseRawInterfaces(json: string | null | undefined): ParsedRosIface[] { export function parseRawInterfaces(json: unknown): ParsedRosIface[] {
if (!json) return [] if (json == null || json === "") return []
try { try {
const parsed = JSON.parse(json) as unknown const parsed = typeof json === "string" ? JSON.parse(json) as unknown : json
const arr = Array.isArray(parsed) ? parsed : [] const arr = Array.isArray(parsed) ? parsed : []
const out: ParsedRosIface[] = [] const out: ParsedRosIface[] = []
for (const item of arr) { for (const item of arr) {
@@ -56,5 +56,5 @@ export function isUniqueConstraintError(err: unknown): boolean {
const rec = err as { code?: unknown; message?: unknown } const rec = err as { code?: unknown; message?: unknown }
const code = String(rec.code ?? "") const code = String(rec.code ?? "")
const msg = String(rec.message ?? "") const msg = String(rec.message ?? "")
return code.includes("SQLITE_CONSTRAINT") || /unique constraint/i.test(msg) return code === "23505" || code.includes("SQLITE_CONSTRAINT") || /unique constraint/i.test(msg)
} }
@@ -6,56 +6,59 @@ import { invalidateFlowCatalogCache } from "../../../services/traffic-flow-topol
export type AppUserRow = typeof appUsers.$inferSelect export type AppUserRow = typeof appUsers.$inferSelect
export type BindingRow = typeof userInterfaceBindings.$inferSelect export type BindingRow = typeof userInterfaceBindings.$inferSelect
export function listUserRows(): AppUserRow[] { export async function listUserRows(): Promise<AppUserRow[]> {
return db.select().from(appUsers).all() return await db.select().from(appUsers)
} }
export function getUserRowById(id: string): AppUserRow | undefined { export async function getUserRowById(id: string): Promise<AppUserRow | undefined> {
return db.select().from(appUsers).where(eq(appUsers.id, id)).limit(1).all()[0] const rows = await db.select().from(appUsers).where(eq(appUsers.id, id)).limit(1)
return rows[0]
} }
export function getUserRowByLogin(login: string): AppUserRow | undefined { export async function getUserRowByLogin(login: string): Promise<AppUserRow | undefined> {
return db.select().from(appUsers).where(eq(appUsers.login, login)).limit(1).all()[0] const rows = await db.select().from(appUsers).where(eq(appUsers.login, login)).limit(1)
return rows[0]
} }
export function createUserRow(values: typeof appUsers.$inferInsert): AppUserRow { export async function createUserRow(values: typeof appUsers.$inferInsert): Promise<AppUserRow> {
const [inserted] = db.insert(appUsers).values(values).returning().all() const [inserted] = await db.insert(appUsers).values(values).returning()
invalidateFlowCatalogCache() invalidateFlowCatalogCache()
return inserted return inserted
} }
export function updateUserRowById( export async function updateUserRowById(
id: string, id: string,
values: Partial<AppUserRow>, values: Partial<AppUserRow>,
): AppUserRow { ): Promise<AppUserRow> {
const [updated] = db.update(appUsers).set(values).where(eq(appUsers.id, id)).returning().all() const [updated] = await db.update(appUsers).set(values).where(eq(appUsers.id, id)).returning()
invalidateFlowCatalogCache() invalidateFlowCatalogCache()
return updated return updated
} }
export function deleteUserRowById(id: string): void { export async function deleteUserRowById(id: string): Promise<void> {
db.delete(appUsers).where(eq(appUsers.id, id)).run() await db.delete(appUsers).where(eq(appUsers.id, id))
invalidateFlowCatalogCache() invalidateFlowCatalogCache()
} }
export function listBindingRows(): BindingRow[] { export async function listBindingRows(): Promise<BindingRow[]> {
return db.select().from(userInterfaceBindings).all() return await db.select().from(userInterfaceBindings)
} }
export function listBindingRowsByUser(userId: string): BindingRow[] { export async function listBindingRowsByUser(userId: string): Promise<BindingRow[]> {
return db.select().from(userInterfaceBindings).where(eq(userInterfaceBindings.userId, userId)).all() return await db.select().from(userInterfaceBindings).where(eq(userInterfaceBindings.userId, userId))
} }
export function getBindingRowById(id: string): BindingRow | undefined { export async function getBindingRowById(id: string): Promise<BindingRow | undefined> {
return db.select().from(userInterfaceBindings).where(eq(userInterfaceBindings.id, id)).limit(1).all()[0] const rows = await db.select().from(userInterfaceBindings).where(eq(userInterfaceBindings.id, id)).limit(1)
return rows[0]
} }
export function getBindingByServerIfacePeer( export async function getBindingByServerIfacePeer(
serverId: number, serverId: number,
interfaceName: string, interfaceName: string,
peerPublicKey = "", peerPublicKey = "",
): BindingRow | undefined { ): Promise<BindingRow | undefined> {
return db const rows = await db
.select() .select()
.from(userInterfaceBindings) .from(userInterfaceBindings)
.where(and( .where(and(
@@ -64,20 +67,21 @@ export function getBindingByServerIfacePeer(
eq(userInterfaceBindings.peerPublicKey, peerPublicKey), eq(userInterfaceBindings.peerPublicKey, peerPublicKey),
)) ))
.limit(1) .limit(1)
.all()[0] return rows[0]
} }
export function createBindingRow(values: typeof userInterfaceBindings.$inferInsert): BindingRow { export async function createBindingRow(values: typeof userInterfaceBindings.$inferInsert): Promise<BindingRow> {
const [inserted] = db.insert(userInterfaceBindings).values(values).returning().all() const [inserted] = await db.insert(userInterfaceBindings).values(values).returning()
invalidateFlowCatalogCache() invalidateFlowCatalogCache()
return inserted return inserted
} }
export function deleteBindingRowById(id: string): void { export async function deleteBindingRowById(id: string): Promise<void> {
db.delete(userInterfaceBindings).where(eq(userInterfaceBindings.id, id)).run() await db.delete(userInterfaceBindings).where(eq(userInterfaceBindings.id, id))
invalidateFlowCatalogCache() invalidateFlowCatalogCache()
} }
export function countUserRows(): number { export async function countUserRows(): Promise<number> {
return db.select({ n: count() }).from(appUsers).all()[0]?.n ?? 0 const rows = await db.select({ n: count() }).from(appUsers)
return rows[0]?.n ?? 0
} }
@@ -12,6 +12,7 @@ import type {
UserBindingCreate, UserBindingCreate,
} from "@mmapp/contracts/users" } from "@mmapp/contracts/users"
import { db } from "../../../db/index.js" import { db } from "../../../db/index.js"
import { parseJsonArray } from "../../../db/json.js"
import { servers, trafficSamples } from "../../../db/schema.js" import { servers, trafficSamples } from "../../../db/schema.js"
import { import {
createBindingRow, createBindingRow,
@@ -52,13 +53,9 @@ export class UsersServiceError extends Error {
} }
} }
function parseJsonArray<T>(raw: string, fallback: T[]): T[] { function asPermArray<T>(raw: unknown, fallback: T[]): T[] {
try { const arr = parseJsonArray(raw)
const parsed = JSON.parse(raw) as unknown return arr.length ? arr as T[] : fallback
return Array.isArray(parsed) ? (parsed as T[]) : fallback
} catch {
return fallback
}
} }
function initials(name: string): string { function initials(name: string): string {
@@ -66,8 +63,9 @@ function initials(name: string): string {
return parts.map((p) => p[0] ?? "").slice(0, 2).join("").toUpperCase() || "??" return parts.map((p) => p[0] ?? "").slice(0, 2).join("").toUpperCase() || "??"
} }
function serverMeta(serverId: number): { name: string; site: string; country: string } { async function serverMeta(serverId: number): Promise<{ name: string; site: string; country: string }> {
const row = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const rows = await db.select().from(servers).where(eq(servers.id, serverId)).limit(1)
const row = rows[0]
return { return {
name: row?.name || row?.host || String(serverId), name: row?.name || row?.host || String(serverId),
site: row?.site || "—", site: row?.site || "—",
@@ -75,8 +73,8 @@ function serverMeta(serverId: number): { name: string; site: string; country: st
} }
} }
function toBindingDto(row: BindingRow): UserBinding { async function toBindingDto(row: BindingRow): Promise<UserBinding> {
const meta = serverMeta(row.serverId) const meta = await serverMeta(row.serverId)
return { return {
id: row.id, id: row.id,
userId: row.userId, userId: row.userId,
@@ -94,7 +92,7 @@ function toBindingDto(row: BindingRow): UserBinding {
} }
} }
function toUserDto(row: AppUserRow, bindings: BindingRow[]): AppUserRead { async function toUserDto(row: AppUserRow, bindings: BindingRow[]): Promise<AppUserRead> {
return { return {
id: row.id, id: row.id,
name: row.name, name: row.name,
@@ -104,39 +102,39 @@ function toUserDto(row: AppUserRow, bindings: BindingRow[]): AppUserRead {
active: Boolean(row.active), active: Boolean(row.active),
avatar: row.avatar, avatar: row.avatar,
lastSeen: row.lastSeen ?? null, lastSeen: row.lastSeen ?? null,
sections: parseJsonArray<SectionPerm>(row.sectionsJson, []), sections: asPermArray<SectionPerm>(row.sectionsJson, []),
servers: parseJsonArray<ServerPerm>(row.serversJson, []), servers: asPermArray<ServerPerm>(row.serversJson, []),
bindings: bindings.map(toBindingDto), bindings: await Promise.all(bindings.map((b) => toBindingDto(b))),
createdAt: row.createdAt, createdAt: row.createdAt,
updatedAt: row.updatedAt, updatedAt: row.updatedAt,
} }
} }
export function listUsers(): AppUserRead[] { export async function listUsers(): Promise<AppUserRead[]> {
const users = listUserRows() const users = await listUserRows()
const allBindings = listBindingRows() const allBindings = await listBindingRows()
const byUser = new Map<string, BindingRow[]>() const byUser = new Map<string, BindingRow[]>()
for (const b of allBindings) { for (const b of allBindings) {
const arr = byUser.get(b.userId) ?? [] const arr = byUser.get(b.userId) ?? []
arr.push(b) arr.push(b)
byUser.set(b.userId, arr) byUser.set(b.userId, arr)
} }
return users.map((u) => toUserDto(u, byUser.get(u.id) ?? [])) return Promise.all(users.map((u) => toUserDto(u, byUser.get(u.id) ?? [])))
} }
export function getUserById(id: string): AppUserRead | undefined { export async function getUserById(id: string): Promise<AppUserRead | undefined> {
const row = getUserRowById(id) const row = await getUserRowById(id)
if (!row) return undefined if (!row) return undefined
return toUserDto(row, listBindingRowsByUser(id)) return await toUserDto(row, await listBindingRowsByUser(id))
} }
export function createUser(input: AppUserCreate): AppUserRead { export async function createUser(input: AppUserCreate): Promise<AppUserRead> {
const login = input.login.trim() const login = input.login.trim()
if (getUserRowByLogin(login)) { if (await getUserRowByLogin(login)) {
throw new UsersServiceError("Логин уже занят", 409) throw new UsersServiceError("Логин уже занят", 409)
} }
const now = new Date().toISOString() const now = new Date().toISOString()
const row = createUserRow({ const row = await createUserRow({
id: randomUUID(), id: randomUUID(),
name: input.name.trim(), name: input.name.trim(),
login, login,
@@ -145,19 +143,19 @@ export function createUser(input: AppUserCreate): AppUserRead {
active: input.active ?? true, active: input.active ?? true,
avatar: (input.avatar ?? "").trim() || initials(input.name), avatar: (input.avatar ?? "").trim() || initials(input.name),
lastSeen: input.lastSeen ?? null, lastSeen: input.lastSeen ?? null,
sectionsJson: JSON.stringify(input.sections ?? []), sectionsJson: input.sections ?? [],
serversJson: JSON.stringify(input.servers ?? []), serversJson: input.servers ?? [],
createdAt: now, createdAt: now,
updatedAt: now, updatedAt: now,
}) })
return toUserDto(row, []) return await toUserDto(row, [])
} }
export function updateUser(id: string, input: AppUserUpdate): AppUserRead { export async function updateUser(id: string, input: AppUserUpdate): Promise<AppUserRead> {
const existing = getUserRowById(id) const existing = await getUserRowById(id)
if (!existing) throw new UsersServiceError("Пользователь не найден", 404) if (!existing) throw new UsersServiceError("Пользователь не найден", 404)
if (input.login != null) { if (input.login != null) {
const other = getUserRowByLogin(input.login.trim()) const other = await getUserRowByLogin(input.login.trim())
if (other && other.id !== id) throw new UsersServiceError("Логин уже занят", 409) if (other && other.id !== id) throw new UsersServiceError("Логин уже занят", 409)
} }
const patch: Partial<AppUserRow> = { updatedAt: new Date().toISOString() } const patch: Partial<AppUserRow> = { updatedAt: new Date().toISOString() }
@@ -168,26 +166,26 @@ export function updateUser(id: string, input: AppUserUpdate): AppUserRead {
if (input.active != null) patch.active = input.active if (input.active != null) patch.active = input.active
if (input.avatar != null) patch.avatar = input.avatar.trim() || existing.avatar if (input.avatar != null) patch.avatar = input.avatar.trim() || existing.avatar
if (input.lastSeen !== undefined) patch.lastSeen = input.lastSeen if (input.lastSeen !== undefined) patch.lastSeen = input.lastSeen
if (input.sections != null) patch.sectionsJson = JSON.stringify(input.sections) if (input.sections != null) patch.sectionsJson = input.sections
if (input.servers != null) patch.serversJson = JSON.stringify(input.servers) if (input.servers != null) patch.serversJson = input.servers
const updated = updateUserRowById(id, patch) const updated = await updateUserRowById(id, patch)
return toUserDto(updated, listBindingRowsByUser(id)) return await toUserDto(updated, await listBindingRowsByUser(id))
} }
export function deleteUser(id: string): void { export async function deleteUser(id: string): Promise<void> {
const existing = getUserRowById(id) const existing = await getUserRowById(id)
if (!existing) throw new UsersServiceError("Пользователь не найден", 404) if (!existing) throw new UsersServiceError("Пользователь не найден", 404)
deleteUserRowById(id) await deleteUserRowById(id)
} }
export function addBinding(userId: string, input: UserBindingCreate): UserBinding { export async function addBinding(userId: string, input: UserBindingCreate): Promise<UserBinding> {
const user = getUserRowById(userId) const user = await getUserRowById(userId)
if (!user) throw new UsersServiceError("Пользователь не найден", 404) if (!user) throw new UsersServiceError("Пользователь не найден", 404)
const server = db.select().from(servers).where(eq(servers.id, input.serverId)).limit(1).all()[0] const serverRows = await db.select().from(servers).where(eq(servers.id, input.serverId)).limit(1)
if (!server) throw new UsersServiceError("Сервер не найден", 404) if (!serverRows[0]) throw new UsersServiceError("Сервер не найден", 404)
const ifaceName = input.interfaceName.trim() const ifaceName = input.interfaceName.trim()
if (!ifaceName) throw new UsersServiceError("Имя интерфейса обязательно", 400) if (!ifaceName) throw new UsersServiceError("Имя интерфейса обязательно", 400)
const type: InterfaceType = input.interfaceType ?? inferIfaceType(input.serverId, ifaceName) const type: InterfaceType = input.interfaceType ?? await inferIfaceType(input.serverId, ifaceName)
let peerPublicKey = "" let peerPublicKey = ""
try { try {
peerPublicKey = normalizeBindingPeer(type, input.peerPublicKey) peerPublicKey = normalizeBindingPeer(type, input.peerPublicKey)
@@ -201,7 +199,7 @@ export function addBinding(userId: string, input: UserBindingCreate): UserBindin
name: input.peerName, name: input.peerName,
}) })
: "" : ""
const taken = getBindingByServerIfacePeer(input.serverId, ifaceName, peerPublicKey) const taken = await getBindingByServerIfacePeer(input.serverId, ifaceName, peerPublicKey)
if (taken) { if (taken) {
throw new UsersServiceError( throw new UsersServiceError(
type === "wg" type === "wg"
@@ -212,7 +210,7 @@ export function addBinding(userId: string, input: UserBindingCreate): UserBindin
} }
const now = new Date().toISOString() const now = new Date().toISOString()
try { try {
const row = createBindingRow({ const row = await createBindingRow({
id: randomUUID(), id: randomUUID(),
userId, userId,
serverId: input.serverId, serverId: input.serverId,
@@ -224,7 +222,7 @@ export function addBinding(userId: string, input: UserBindingCreate): UserBindin
createdAt: now, createdAt: now,
updatedAt: now, updatedAt: now,
}) })
return toBindingDto(row) return await toBindingDto(row)
} catch (err) { } catch (err) {
if (isUniqueConstraintError(err)) { if (isUniqueConstraintError(err)) {
throw new UsersServiceError( throw new UsersServiceError(
@@ -238,37 +236,37 @@ export function addBinding(userId: string, input: UserBindingCreate): UserBindin
} }
} }
export function removeBinding(userId: string, bindingId: string): void { export async function removeBinding(userId: string, bindingId: string): Promise<void> {
const row = getBindingRowById(bindingId) const row = await getBindingRowById(bindingId)
if (!row || row.userId !== userId) { if (!row || row.userId !== userId) {
throw new UsersServiceError("Привязка не найдена", 404) throw new UsersServiceError("Привязка не найдена", 404)
} }
deleteBindingRowById(bindingId) await deleteBindingRowById(bindingId)
} }
function inferIfaceType(serverId: number, ifaceName: string): InterfaceType { async function inferIfaceType(serverId: number, ifaceName: string): Promise<InterfaceType> {
const snap = getLatestSnapshot(serverId) const snap = await getLatestSnapshot(serverId)
const parsed = parseRawInterfaces(snap?.rawInterfaces) const parsed = parseRawInterfaces(snap?.rawInterfaces)
const found = parsed.find((i) => i.name === ifaceName) const found = parsed.find((i) => i.name === ifaceName)
return found?.type ?? "other" return found?.type ?? "other"
} }
export async function listInterfaceCatalog(serverId: number): Promise<CatalogInterface[]> { export async function listInterfaceCatalog(serverId: number): Promise<CatalogInterface[]> {
const server = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const serverRows = await db.select().from(servers).where(eq(servers.id, serverId)).limit(1)
if (!server) throw new UsersServiceError("Сервер не найден", 404) if (!serverRows[0]) throw new UsersServiceError("Сервер не найден", 404)
const snap = getLatestSnapshot(serverId) const snap = await getLatestSnapshot(serverId)
let ifaces = parseRawInterfaces(snap?.rawInterfaces) let ifaces = parseRawInterfaces(snap?.rawInterfaces)
if (ifaces.length === 0) { if (ifaces.length === 0) {
const last = db const lastRows = await db
.select({ sampledAt: trafficSamples.sampledAt }) .select({ sampledAt: trafficSamples.sampledAt })
.from(trafficSamples) .from(trafficSamples)
.where(eq(trafficSamples.serverId, serverId)) .where(eq(trafficSamples.serverId, serverId))
.orderBy(desc(trafficSamples.sampledAt)) .orderBy(desc(trafficSamples.sampledAt))
.limit(1) .limit(1)
.all()[0] const last = lastRows[0]
if (last) { if (last) {
const rows = db const rows = (await db
.select({ .select({
interfaceName: trafficSamples.interfaceName, interfaceName: trafficSamples.interfaceName,
peerPublicKey: trafficSamples.peerPublicKey, peerPublicKey: trafficSamples.peerPublicKey,
@@ -276,8 +274,7 @@ export async function listInterfaceCatalog(serverId: number): Promise<CatalogInt
disabled: trafficSamples.disabled, disabled: trafficSamples.disabled,
}) })
.from(trafficSamples) .from(trafficSamples)
.where(eq(trafficSamples.serverId, serverId)) .where(eq(trafficSamples.serverId, serverId)))
.all()
.filter((r) => r.interfaceName && !/^(lo|loopback)/i.test(r.interfaceName) && !(r.peerPublicKey ?? "")) .filter((r) => r.interfaceName && !/^(lo|loopback)/i.test(r.interfaceName) && !(r.peerPublicKey ?? ""))
const seen = new Set<string>() const seen = new Set<string>()
ifaces = [] ifaces = []
@@ -294,8 +291,8 @@ export async function listInterfaceCatalog(serverId: number): Promise<CatalogInt
} }
} }
const bindings = listBindingRows().filter((b) => b.serverId === serverId) const bindings = (await listBindingRows()).filter((b) => b.serverId === serverId)
const usersById = new Map(listUserRows().map((u) => [u.id, u])) const usersById = new Map((await listUserRows()).map((u) => [u.id, u]))
const hasWg = ifaces.some((i) => i.type === "wg") const hasWg = ifaces.some((i) => i.type === "wg")
const wgLive = hasWg const wgLive = hasWg
? await listWireGuardPeersForCatalog(serverId) ? await listWireGuardPeersForCatalog(serverId)
-1
View File
@@ -10,7 +10,6 @@ process.env.AUTH_JWT_SECRET = "test-secret-at-least-8"
process.env.AUTH_ISSUER = "https://auth.test.local" process.env.AUTH_ISSUER = "https://auth.test.local"
process.env.AUTH_PORTAL_URL = "http://localhost:5175" process.env.AUTH_PORTAL_URL = "http://localhost:5175"
process.env.CORS_ORIGIN = "http://localhost:3000" process.env.CORS_ORIGIN = "http://localhost:3000"
process.env.DATABASE_PATH = ":memory:"
process.env.NODE_ENV = "test" process.env.NODE_ENV = "test"
// Dynamic import after env is set // Dynamic import after env is set
+12 -12
View File
@@ -20,14 +20,14 @@ import {
const alertsRoutes: FastifyPluginAsyncZod = async (app) => { const alertsRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/alerts", async (_req, reply) => { app.get("/alerts", async (_req, reply) => {
const tg = getTelegramPublic() const tg = await getTelegramPublic()
const connected = tg.tokenConfigured && Boolean(tg.chatId?.trim()) const connected = tg.tokenConfigured && Boolean(tg.chatId?.trim())
return reply.send({ return reply.send({
telegram: { ...tg, connected }, telegram: { ...tg, connected },
groups: listAlertGroups(), groups: await listAlertGroups(),
rules: listAlertRules(), rules: await listAlertRules(),
history: listMergedHistory(), history: await listMergedHistory(),
meta: getAlertsMeta(), meta: await getAlertsMeta(),
}) })
}) })
@@ -38,7 +38,7 @@ const alertsRoutes: FastifyPluginAsyncZod = async (app) => {
} }
const groups = const groups =
parsed.data.groups ?? parsed.data.groups ??
listAlertGroups().map((g) => ({ (await listAlertGroups()).map((g) => ({
id: g.id, id: g.id,
name: g.name, name: g.name,
combineMode: g.combineMode, combineMode: g.combineMode,
@@ -67,8 +67,8 @@ const alertsRoutes: FastifyPluginAsyncZod = async (app) => {
recoveryStabilitySec: r.recoveryStabilitySec ?? null, recoveryStabilitySec: r.recoveryStabilitySec ?? null,
chatId: r.chatId, chatId: r.chatId,
})) }))
replaceAlertsConfig({ groups, rules }) await replaceAlertsConfig({ groups, rules })
return reply.send({ ok: true, groups: listAlertGroups(), rules: listAlertRules() }) return reply.send({ ok: true, groups: await listAlertGroups(), rules: await listAlertRules() })
}) })
app.put("/alerts/telegram", async (req, reply) => { app.put("/alerts/telegram", async (req, reply) => {
@@ -76,7 +76,7 @@ const alertsRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const next = updateTelegramSettings({ const next = await updateTelegramSettings({
token: parsed.data.token, token: parsed.data.token,
chatId: parsed.data.chatId, chatId: parsed.data.chatId,
messageThreadId: parsed.data.messageThreadId, messageThreadId: parsed.data.messageThreadId,
@@ -90,9 +90,9 @@ const alertsRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const fromDb = getTelegramBotToken() const fromDb = await getTelegramBotToken()
const token = (parsed.data.token?.trim() || fromDb).trim() const token = (parsed.data.token?.trim() || fromDb).trim()
const pub = getTelegramPublic() const pub = await getTelegramPublic()
const chatId = (parsed.data.chatId?.trim() || pub.chatId || "").trim() const chatId = (parsed.data.chatId?.trim() || pub.chatId || "").trim()
if (!token) { if (!token) {
return reply.status(400).send({ error: "Не задан Bot Token (сохраните в БД или передайте в запросе)" }) return reply.status(400).send({ error: "Не задан Bot Token (сохраните в БД или передайте в запросе)" })
@@ -103,7 +103,7 @@ const alertsRoutes: FastifyPluginAsyncZod = async (app) => {
const threadId = const threadId =
parsed.data.messageThreadId != null && parsed.data.messageThreadId >= 1 parsed.data.messageThreadId != null && parsed.data.messageThreadId >= 1
? parsed.data.messageThreadId ? parsed.data.messageThreadId
: getTelegramMessageThreadIdForApi() : await getTelegramMessageThreadIdForApi()
const text = parsed.data.rulePreview const text = parsed.data.rulePreview
? formatAlertRuleTestTelegramText(parsed.data.rulePreview) ? formatAlertRuleTestTelegramText(parsed.data.rulePreview)
: "MikroTik Manager — тест оповещений (Telegram)." : "MikroTik Manager — тест оповещений (Telegram)."
+11 -11
View File
@@ -61,7 +61,7 @@ async function processBackupJob(job: BackupJob, ids: string[], notes?: string) {
} }
job.status = "done" job.status = "done"
job.finishedAt = new Date().toISOString() job.finishedAt = new Date().toISOString()
appendEvent({ await appendEvent({
level: job.failures.length > 0 ? "warning" : "info", level: job.failures.length > 0 ? "warning" : "info",
eventType: "backups.job.done", eventType: "backups.job.done",
sourceModule: "backups", sourceModule: "backups",
@@ -80,11 +80,11 @@ async function processBackupJob(job: BackupJob, ids: string[], notes?: string) {
const backupsRoutes: FastifyPluginAsyncZod = async (app) => { const backupsRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/backups", async (_req, reply) => { app.get("/backups", async (_req, reply) => {
return reply.send(listBackups()) return reply.send(await listBackups())
}) })
app.get("/backups/schedule", async (_req, reply) => { app.get("/backups/schedule", async (_req, reply) => {
return reply.send(getBackupScheduleSettings()) return reply.send(await getBackupScheduleSettings())
}) })
app.put("/backups/schedule", async (req, reply) => { app.put("/backups/schedule", async (req, reply) => {
@@ -92,15 +92,15 @@ const backupsRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const result = updateBackupScheduleSettings(parsed.data) const result = await updateBackupScheduleSettings(parsed.data)
refreshScheduler() await refreshScheduler()
return reply.send(result) return reply.send(result)
}) })
app.post("/backups/create", { schema: { body: CreateBackupBodySchema } }, async (req, reply) => { app.post("/backups/create", { schema: { body: CreateBackupBodySchema } }, async (req, reply) => {
const inputIds = req.body.serverIds.map((x) => String(x)) const inputIds = req.body.serverIds.map((x) => String(x))
const notes = req.body.notes?.trim() || undefined const notes = req.body.notes?.trim() || undefined
const existingServers = new Set(listServersRead().map((s) => String(s.id))) const existingServers = new Set((await listServersRead()).map((s) => String(s.id)))
const ids = [...new Set(inputIds)].filter((id) => existingServers.has(id)) const ids = [...new Set(inputIds)].filter((id) => existingServers.has(id))
if (ids.length === 0) return reply.status(400).send({ error: "Не выбраны валидные серверы" }) if (ids.length === 0) return reply.status(400).send({ error: "Не выбраны валидные серверы" })
@@ -115,7 +115,7 @@ const backupsRoutes: FastifyPluginAsyncZod = async (app) => {
failures: [], failures: [],
} }
backupJobs.set(jobId, job) backupJobs.set(jobId, job)
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "backups.job.started", eventType: "backups.job.started",
sourceModule: "backups", sourceModule: "backups",
@@ -128,8 +128,8 @@ const backupsRoutes: FastifyPluginAsyncZod = async (app) => {
notes: notes ?? null, notes: notes ?? null,
}, },
}) })
queueMicrotask(() => { queueMicrotask(async () => {
void processBackupJob(job, ids, notes).catch((err) => { void processBackupJob(job, ids, notes).catch(async (err) => {
job.status = "failed" job.status = "failed"
job.finishedAt = new Date().toISOString() job.finishedAt = new Date().toISOString()
const failure = { const failure = {
@@ -137,7 +137,7 @@ const backupsRoutes: FastifyPluginAsyncZod = async (app) => {
error: err instanceof Error ? err.message : String(err), error: err instanceof Error ? err.message : String(err),
} }
job.failures.push(failure) job.failures.push(failure)
appendEvent({ await appendEvent({
level: "critical", level: "critical",
eventType: "backups.job.failed", eventType: "backups.job.failed",
sourceModule: "backups", sourceModule: "backups",
@@ -167,7 +167,7 @@ const backupsRoutes: FastifyPluginAsyncZod = async (app) => {
}) })
app.get("/backups/:id/download", { schema: { params: BackupIdParamSchema } }, async (req, reply) => { app.get("/backups/:id/download", { schema: { params: BackupIdParamSchema } }, async (req, reply) => {
const hit = getBackupById(req.params.id) const hit = await getBackupById(req.params.id)
if (!hit) return reply.status(404).send({ error: "Бэкап не найден" }) if (!hit) return reply.status(404).send({ error: "Бэкап не найден" })
const filePath = path.join(getBackupsDir(), hit.filename) const filePath = path.join(getBackupsDir(), hit.filename)
const content = await readFile(filePath, "utf8").catch(() => null) const content = await readFile(filePath, "utf8").catch(() => null)
+4 -4
View File
@@ -12,7 +12,7 @@ const bgpRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/bgp/sessions/raw/:id — raw RouterOS response for a single server (debug) // GET /api/bgp/sessions/raw/:id — raw RouterOS response for a single server (debug)
app.get("/bgp/sessions/raw/:id", { schema: { params: ServerIdParamSchema } }, async (req, reply) => { app.get("/bgp/sessions/raw/:id", { schema: { params: ServerIdParamSchema } }, async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const server = db.select().from(servers).where(eq(servers.id, params.id)).limit(1).all()[0] const server = (await db.select().from(servers).where(eq(servers.id, params.id)).limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
try { try {
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
@@ -26,7 +26,7 @@ const bgpRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/bgp/sessions — aggregate from ALL enabled servers // GET /api/bgp/sessions — aggregate from ALL enabled servers
app.get("/bgp/sessions", async (_req, reply) => { app.get("/bgp/sessions", async (_req, reply) => {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const results: BgpSessionRead[][] = await Promise.all( const results: BgpSessionRead[][] = await Promise.all(
allServers.map(async (server) => { allServers.map(async (server) => {
@@ -46,10 +46,10 @@ const bgpRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/servers/:id/bgp/sessions — single server // GET /api/servers/:id/bgp/sessions — single server
app.get("/servers/:id/bgp/sessions", { schema: { params: ServerIdParamSchema } }, async (req, reply) => { app.get("/servers/:id/bgp/sessions", { schema: { params: ServerIdParamSchema } }, async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const server = db const server = (await db
.select().from(servers) .select().from(servers)
.where(eq(servers.id, params.id)) .where(eq(servers.id, params.id))
.limit(1).all()[0] .limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
+10 -10
View File
@@ -32,7 +32,7 @@ const certificatesRoutes: FastifyPluginAsyncZod = async (app) => {
}) })
app.get("/certificates/acme-settings", async (_req, reply) => { app.get("/certificates/acme-settings", async (_req, reply) => {
return reply.send(getAcmeSettingsPublic()) return reply.send(await getAcmeSettingsPublic())
}) })
app.put("/certificates/acme-settings", async (req, reply) => { app.put("/certificates/acme-settings", async (req, reply) => {
@@ -40,11 +40,11 @@ const certificatesRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
return reply.send(updateAcmeSettings(parsed.data)) return reply.send(await updateAcmeSettings(parsed.data))
}) })
app.get("/certificates/renew-settings", async (_req, reply) => { app.get("/certificates/renew-settings", async (_req, reply) => {
return reply.send(getCertificateRenewSettings()) return reply.send(await getCertificateRenewSettings())
}) })
app.put("/certificates/renew-settings", async (req, reply) => { app.put("/certificates/renew-settings", async (req, reply) => {
@@ -52,8 +52,8 @@ const certificatesRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const result = updateCertificateRenewSettings(parsed.data) const result = await updateCertificateRenewSettings(parsed.data)
refreshScheduler() await refreshScheduler()
return reply.send(result) return reply.send(result)
}) })
@@ -62,7 +62,7 @@ const certificatesRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const effective = parsed.data.cloudflareApiToken?.trim() || getAcmeCloudflareToken() const effective = parsed.data.cloudflareApiToken?.trim() || await getAcmeCloudflareToken()
if (!effective) { if (!effective) {
return reply.status(400).send({ error: "Не задан Cloudflare API token" }) return reply.status(400).send({ error: "Не задан Cloudflare API token" })
} }
@@ -83,12 +83,12 @@ const certificatesRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const serverId = String(parsed.data.serverId) const serverId = String(parsed.data.serverId)
const server = getServerRowByIdString(serverId) const server = await getServerRowByIdString(serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const jobId = randomUUID() const jobId = randomUUID()
const trustStore = (parsed.data.trustStore ?? ["www", "api"]).join(",") const trustStore = (parsed.data.trustStore ?? ["www", "api"]).join(",")
createIssueJobRecord({ await createIssueJobRecord({
id: jobId, id: jobId,
serverId, serverId,
certName: parsed.data.certName.trim(), certName: parsed.data.certName.trim(),
@@ -103,9 +103,9 @@ const certificatesRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/certificates/issue/:jobId", async (req, reply) => { app.get("/certificates/issue/:jobId", async (req, reply) => {
const jobId = String((req.params as { jobId: string }).jobId) const jobId = String((req.params as { jobId: string }).jobId)
const row = getIssueJobRecord(jobId) const row = await getIssueJobRecord(jobId)
if (!row) return reply.status(404).send({ error: "Задача не найдена" }) if (!row) return reply.status(404).send({ error: "Задача не найдена" })
return reply.send(toIssueJobDto(row)) return reply.send(await toIssueJobDto(row))
}) })
} }
+2 -2
View File
@@ -8,7 +8,7 @@ const eventsRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректные параметры запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректные параметры запроса", details: parsed.error.flatten() })
} }
return reply.send({ events: readEvents(parsed.data) }) return reply.send({ events: await readEvents(parsed.data) })
}) })
app.post("/events/batch", async (req, reply) => { app.post("/events/batch", async (req, reply) => {
@@ -16,7 +16,7 @@ const eventsRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
appendEvents(parsed.data.events) await appendEvents(parsed.data.events)
return reply.status(201).send({ ok: true, inserted: parsed.data.events.length }) return reply.status(201).send({ ok: true, inserted: parsed.data.events.length })
}) })
} }
+17 -19
View File
@@ -17,11 +17,11 @@ const TestEvobgpSchema = z.object({
apiKey: z.string().optional(), apiKey: z.string().optional(),
}) })
function ensureEvobgpRow() { async function ensureEvobgpRow() {
let row = db.select().from(evobgpSettings).where(eq(evobgpSettings.id, 1)).limit(1).all()[0] let row = (await db.select().from(evobgpSettings).where(eq(evobgpSettings.id, 1)).limit(1))[0]
if (!row) { if (!row) {
db.insert(evobgpSettings).values({ id: 1 }).run() await db.insert(evobgpSettings).values({ id: 1 })
row = db.select().from(evobgpSettings).where(eq(evobgpSettings.id, 1)).limit(1).all()[0] row = (await db.select().from(evobgpSettings).where(eq(evobgpSettings.id, 1)).limit(1))[0]
} }
return row! return row!
} }
@@ -108,11 +108,10 @@ function aggregateCommunityRouteCounts(catalog: EvoCatalogRaw): Map<string, numb
* Уникальные серверы из локальной БД, у которых в BGP-фильтре указан community (строка AS:NNN). * Уникальные серверы из локальной БД, у которых в BGP-фильтре указан community (строка AS:NNN).
* Ключ — значение `filter_rules.community` после trim (как в EvoBGP `community`). * Ключ — значение `filter_rules.community` после trim (как в EvoBGP `community`).
*/ */
function distinctServersByFilterCommunityValue(): Map<string, number> { async function distinctServersByFilterCommunityValue(): Promise<Map<string, number>> {
const rows = db const rows = await db
.select({ serverId: filterRules.serverId, community: filterRules.community }) .select({ serverId: filterRules.serverId, community: filterRules.community })
.from(filterRules) .from(filterRules)
.all()
const byVal = new Map<string, Set<number>>() const byVal = new Map<string, Set<number>>()
for (const r of rows) { for (const r of rows) {
const v = (r.community ?? "").trim() const v = (r.community ?? "").trim()
@@ -161,8 +160,8 @@ async function fetchEvoJson<T>(root: string, path: string, token: string): Promi
} }
} }
function credentialsFromDb(): { root: string; apiKey: string } | null { async function credentialsFromDb(): Promise<{ root: string; apiKey: string } | null> {
const row = ensureEvobgpRow() const row = await ensureEvobgpRow()
const root = normalizeBaseUrl(row.baseUrl) const root = normalizeBaseUrl(row.baseUrl)
const apiKey = normalizeApiKey(row.apiKey) const apiKey = normalizeApiKey(row.apiKey)
if (!root || !apiKey) return null if (!root || !apiKey) return null
@@ -171,7 +170,7 @@ function credentialsFromDb(): { root: string; apiKey: string } | null {
const evobgpRoutes: FastifyPluginAsyncZod = async (app) => { const evobgpRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/evobgp/settings", async (_req, reply) => { app.get("/evobgp/settings", async (_req, reply) => {
const row = ensureEvobgpRow() const row = await ensureEvobgpRow()
return reply.send({ return reply.send({
baseUrl: row.baseUrl ?? "", baseUrl: row.baseUrl ?? "",
enabled: Boolean(row.enabled), enabled: Boolean(row.enabled),
@@ -184,7 +183,7 @@ const evobgpRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const cur = ensureEvobgpRow() const cur = await ensureEvobgpRow()
let nextBase = cur.baseUrl let nextBase = cur.baseUrl
let nextEnabled = cur.enabled let nextEnabled = cur.enabled
let nextKey = cur.apiKey let nextKey = cur.apiKey
@@ -200,17 +199,16 @@ const evobgpRoutes: FastifyPluginAsyncZod = async (app) => {
: normalizeApiKey(parsed.data.apiKey) : normalizeApiKey(parsed.data.apiKey)
} }
db.update(evobgpSettings) await db.update(evobgpSettings)
.set({ .set({
baseUrl: nextBase, baseUrl: nextBase,
enabled: nextEnabled, enabled: nextEnabled,
apiKey: nextKey, apiKey: nextKey,
updatedAt: sql`(datetime('now'))`, updatedAt: sql`now()`,
}) })
.where(eq(evobgpSettings.id, 1)) .where(eq(evobgpSettings.id, 1))
.run()
const row = ensureEvobgpRow() const row = await ensureEvobgpRow()
return reply.send({ return reply.send({
baseUrl: row.baseUrl ?? "", baseUrl: row.baseUrl ?? "",
enabled: Boolean(row.enabled), enabled: Boolean(row.enabled),
@@ -228,7 +226,7 @@ const evobgpRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const row = ensureEvobgpRow() const row = await ensureEvobgpRow()
const d = parsed.data const d = parsed.data
const urlRaw = d.baseUrl !== undefined ? d.baseUrl : row.baseUrl const urlRaw = d.baseUrl !== undefined ? d.baseUrl : row.baseUrl
const keyRaw = const keyRaw =
@@ -253,11 +251,11 @@ const evobgpRoutes: FastifyPluginAsyncZod = async (app) => {
* Прокси к GET /v1/router-lists/catalog — учётные данные только из БД. * Прокси к GET /v1/router-lists/catalog — учётные данные только из БД.
*/ */
app.post("/evobgp/catalog", async (_req, reply) => { app.post("/evobgp/catalog", async (_req, reply) => {
const row = ensureEvobgpRow() const row = await ensureEvobgpRow()
if (!row.enabled) { if (!row.enabled) {
return reply.status(400).send({ error: "Интеграция EvoBGP выключена в настройках" }) return reply.status(400).send({ error: "Интеграция EvoBGP выключена в настройках" })
} }
const cred = credentialsFromDb() const cred = await credentialsFromDb()
if (!cred) { if (!cred) {
return reply.status(400).send({ error: "Не заданы базовый URL или API-ключ в БД" }) return reply.status(400).send({ error: "Не заданы базовый URL или API-ключ в БД" })
} }
@@ -333,7 +331,7 @@ const evobgpRoutes: FastifyPluginAsyncZod = async (app) => {
}) })
const routeCountsByCommUuid = aggregateCommunityRouteCounts(catalog) const routeCountsByCommUuid = aggregateCommunityRouteCounts(catalog)
const serversByCommunityStr = distinctServersByFilterCommunityValue() const serversByCommunityStr = await distinctServersByFilterCommunityValue()
const communities = commItems.map((c) => { const communities = commItems.map((c) => {
const valueStr = (c.community ?? "").trim() const valueStr = (c.community ?? "").trim()
+2 -2
View File
@@ -314,10 +314,10 @@ const execRoutes: FastifyPluginAsyncZod = async (app) => {
{ schema: { params: ServerIdParamSchema, body: ExecBodySchema } }, { schema: { params: ServerIdParamSchema, body: ExecBodySchema } },
async (req, reply) => { async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const server = db const server = (await db
.select().from(servers) .select().from(servers)
.where(eq(servers.id, params.id)) .where(eq(servers.id, params.id))
.limit(1).all()[0] .limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
+35 -39
View File
@@ -186,12 +186,11 @@ function gatewayFromRecursiveDst(dstAddress: string): string {
* Импорт с роутера даёт только `set gateway` без `set out-interface` — в таком виде не отличить от «поломанного» GRE. * Импорт с роутера даёт только `set gateway` без `set out-interface` — в таком виде не отличить от «поломанного» GRE.
* Сопоставляем hop с локальной таблицей recursive_routes и восстанавливаем gatewayTunnelId = rec:<id>. * Сопоставляем hop с локальной таблицей recursive_routes и восстанавливаем gatewayTunnelId = rec:<id>.
*/ */
function enrichRulesWithRecursiveGateway(serverId: number, rules: ApiFilterRule[]): ApiFilterRule[] { async function enrichRulesWithRecursiveGateway(serverId: number, rules: ApiFilterRule[]): Promise<ApiFilterRule[]> {
const rows = db const rows = await db
.select() .select()
.from(recursiveRoutes) .from(recursiveRoutes)
.where(and(eq(recursiveRoutes.serverId, serverId), eq(recursiveRoutes.disabled, false))) .where(and(eq(recursiveRoutes.serverId, serverId), eq(recursiveRoutes.disabled, false)))
.all()
return rules.map(rule => { return rules.map(rule => {
if (rule.action === "blackhole") return rule if (rule.action === "blackhole") return rule
@@ -249,7 +248,7 @@ async function fetchServerFilters(server: ServerRow) {
.filter(r => (r.chain ?? "").trim().toLowerCase() === "bgp-in") .filter(r => (r.chain ?? "").trim().toLowerCase() === "bgp-in")
.flatMap(parseFilterRule) .flatMap(parseFilterRule)
const rules = enrichRulesWithRecursiveGateway(server.id, rulesRaw) const rules = await enrichRulesWithRecursiveGateway(server.id, rulesRaw)
return { return {
serverId: String(server.id), serverId: String(server.id),
@@ -258,8 +257,8 @@ async function fetchServerFilters(server: ServerRow) {
} }
} }
function toApiRulesets(serverRows: ServerRow[]) { async function toApiRulesets(serverRows: ServerRow[]) {
const dbRules = db.select().from(filterRules).orderBy(asc(filterRules.serverId), asc(filterRules.sortOrder)).all() const dbRules = await db.select().from(filterRules).orderBy(asc(filterRules.serverId), asc(filterRules.sortOrder))
return serverRows.map(s => ({ return serverRows.map(s => ({
serverId: String(s.id), serverId: String(s.id),
rules: dbRules rules: dbRules
@@ -277,7 +276,7 @@ function toApiRulesets(serverRows: ServerRow[]) {
} }
/** GRE: gatewayTunnelId = имя интерфейса; рекурсивный: rec:<id строки recursive_routes */ /** GRE: gatewayTunnelId = имя интерфейса; рекурсивный: rec:<id строки recursive_routes */
function resolveRouteTargets(serverId: number, rule: ApiFilterRule): { gateway: string; outIface: string } { async function resolveRouteTargets(serverId: number, rule: ApiFilterRule): Promise<{ gateway: string; outIface: string }> {
if (rule.action === "blackhole") return { gateway: "", outIface: "" } if (rule.action === "blackhole") return { gateway: "", outIface: "" }
const tid = (rule.gatewayTunnelId ?? "").trim() const tid = (rule.gatewayTunnelId ?? "").trim()
if (tid.startsWith("rec:")) { if (tid.startsWith("rec:")) {
@@ -285,12 +284,11 @@ function resolveRouteTargets(serverId: number, rule: ApiFilterRule): { gateway:
if (!Number.isFinite(rid)) { if (!Number.isFinite(rid)) {
return { gateway: rule.gateway, outIface: "" } return { gateway: rule.gateway, outIface: "" }
} }
const row = db const row = (await db
.select() .select()
.from(recursiveRoutes) .from(recursiveRoutes)
.where(and(eq(recursiveRoutes.serverId, serverId), eq(recursiveRoutes.id, rid))) .where(and(eq(recursiveRoutes.serverId, serverId), eq(recursiveRoutes.id, rid)))
.limit(1) .limit(1))[0]
.all()[0]
if (!row) return { gateway: rule.gateway, outIface: "" } if (!row) return { gateway: rule.gateway, outIface: "" }
const gw = gatewayFromRecursiveDst(row.dstAddress) const gw = gatewayFromRecursiveDst(row.dstAddress)
return { gateway: gw || rule.gateway, outIface: "" } return { gateway: gw || rule.gateway, outIface: "" }
@@ -303,30 +301,30 @@ function normalizeCommunity(c: string): string {
} }
/** Одинаковый эффект на роутере при одинаковой community (blackhole vs gateway + out-interface) */ /** Одинаковый эффект на роутере при одинаковой community (blackhole vs gateway + out-interface) */
function ruleEffectSignature(serverId: number, r: ApiFilterRule): string { async function ruleEffectSignature(serverId: number, r: ApiFilterRule): Promise<string> {
if (r.action === "blackhole") return `bh:${normalizeCommunity(r.community)}` if (r.action === "blackhole") return `bh:${normalizeCommunity(r.community)}`
const { gateway, outIface } = resolveRouteTargets(serverId, r) const { gateway, outIface } = await resolveRouteTargets(serverId, r)
return `rt:${normalizeCommunity(r.community)}:${gateway}:${outIface}` return `rt:${normalizeCommunity(r.community)}:${gateway}:${outIface}`
} }
export type FilterRouterCompareStatus = "synced" | "drift" | "missing" export type FilterRouterCompareStatus = "synced" | "drift" | "missing"
function compareDbRulesWithRouter( async function compareDbRulesWithRouter(
serverId: number, serverId: number,
dbRules: ApiFilterRule[], dbRules: ApiFilterRule[],
remoteRules: ApiFilterRule[], remoteRules: ApiFilterRule[],
): Record<string, FilterRouterCompareStatus> { ): Promise<Record<string, FilterRouterCompareStatus>> {
const remoteSigByComm = new Map<string, string>() const remoteSigByComm = new Map<string, string>()
for (const rr of remoteRules) { for (const rr of remoteRules) {
const c = normalizeCommunity(rr.community) const c = normalizeCommunity(rr.community)
if (!remoteSigByComm.has(c)) { if (!remoteSigByComm.has(c)) {
remoteSigByComm.set(c, ruleEffectSignature(serverId, rr)) remoteSigByComm.set(c, await ruleEffectSignature(serverId, rr))
} }
} }
const out: Record<string, FilterRouterCompareStatus> = {} const out: Record<string, FilterRouterCompareStatus> = {}
for (const dr of dbRules) { for (const dr of dbRules) {
const c = normalizeCommunity(dr.community) const c = normalizeCommunity(dr.community)
const sigD = ruleEffectSignature(serverId, dr) const sigD = await ruleEffectSignature(serverId, dr)
const sigR = remoteSigByComm.get(c) const sigR = remoteSigByComm.get(c)
if (sigR === undefined) { if (sigR === undefined) {
out[c] = "missing" out[c] = "missing"
@@ -339,7 +337,7 @@ function compareDbRulesWithRouter(
return out return out
} }
function toRouterRuleBody(serverId: number, rules: ApiFilterRule[]): string { async function toRouterRuleBody(serverId: number, rules: ApiFilterRule[]): Promise<string> {
if (rules.length === 0) return "" if (rules.length === 0) return ""
// Группируем по эффекту (action + gateway + out-interface). Communities с одним и тем же // Группируем по эффекту (action + gateway + out-interface). Communities с одним и тем же
// `set gw` объединяются через `||` в один if-блок — компактнее и ближе к привычному // `set gw` объединяются через `||` в один if-блок — компактнее и ближе к привычному
@@ -361,7 +359,7 @@ function toRouterRuleBody(serverId: number, rules: ApiFilterRule[]): string {
const isBlackhole = rule.action === "blackhole" const isBlackhole = rule.action === "blackhole"
const { gateway, outIface } = isBlackhole const { gateway, outIface } = isBlackhole
? { gateway: "", outIface: "" } ? { gateway: "", outIface: "" }
: resolveRouteTargets(serverId, rule) : await resolveRouteTargets(serverId, rule)
const key = isBlackhole ? "bh" : `rt:${gateway}:${outIface}` const key = isBlackhole ? "bh" : `rt:${gateway}:${outIface}`
let idx = indexByKey.get(key) let idx = indexByKey.get(key)
if (idx === undefined) { if (idx === undefined) {
@@ -404,10 +402,10 @@ function toRouterRuleBody(serverId: number, rules: ApiFilterRule[]): string {
} }
async function replaceDbRules(serverId: number, rules: ApiFilterRule[]) { async function replaceDbRules(serverId: number, rules: ApiFilterRule[]) {
db.delete(filterRules).where(eq(filterRules.serverId, serverId)).run() await db.delete(filterRules).where(eq(filterRules.serverId, serverId))
if (rules.length === 0) return if (rules.length === 0) return
const now = new Date().toISOString() const now = new Date().toISOString()
db.insert(filterRules).values( await db.insert(filterRules).values(
rules.map((r, i) => ({ rules.map((r, i) => ({
serverId, serverId,
sortOrder: i, sortOrder: i,
@@ -420,7 +418,7 @@ async function replaceDbRules(serverId: number, rules: ApiFilterRule[]) {
createdAt: now, createdAt: now,
updatedAt: now, updatedAt: now,
})), })),
).run() )
} }
const filtersRoutes: FastifyPluginAsyncZod = async (app) => { const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
@@ -432,17 +430,16 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "serverId is required" }) return reply.status(400).send({ error: "serverId is required" })
} }
const server = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const server = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
try { try {
const remote = await fetchServerFilters(server) const remote = await fetchServerFilters(server)
const rows = db const rows = await db
.select() .select()
.from(filterRules) .from(filterRules)
.where(eq(filterRules.serverId, serverId)) .where(eq(filterRules.serverId, serverId))
.orderBy(asc(filterRules.sortOrder)) .orderBy(asc(filterRules.sortOrder))
.all()
const dbRules: ApiFilterRule[] = rows.map(r => ({ const dbRules: ApiFilterRule[] = rows.map(r => ({
id: String(r.id), id: String(r.id),
@@ -454,7 +451,7 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
description: r.description, description: r.description,
})) }))
const byCommunity = compareDbRulesWithRouter(serverId, dbRules, remote.rules) const byCommunity = await compareDbRulesWithRouter(serverId, dbRules, remote.rules)
return reply.send({ byCommunity }) return reply.send({ byCommunity })
} catch (err) { } catch (err) {
app.log.error({ serverId, err: String(err) }, "filters router-compare failed") app.log.error({ serverId, err: String(err) }, "filters router-compare failed")
@@ -466,7 +463,7 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/filters/gre-tunnels", async (req, reply) => { app.get("/filters/gre-tunnels", async (req, reply) => {
const q = req.query as { serverId?: string | number } const q = req.query as { serverId?: string | number }
const sid = parseDbServerId(q.serverId) const sid = parseDbServerId(q.serverId)
const enabledServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const enabledServers = await db.select().from(servers).where(eq(servers.enabled, true))
if (sid !== null) { if (sid !== null) {
const server = enabledServers.find(s => s.id === sid) const server = enabledServers.find(s => s.id === sid)
@@ -492,8 +489,8 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
/** Только правила фильтров из БД (без опроса MikroTik за GRE) */ /** Только правила фильтров из БД (без опроса MikroTik за GRE) */
app.get("/filters/rules", async (_req, reply) => { app.get("/filters/rules", async (_req, reply) => {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const dbRulesets = toApiRulesets(allServers) const dbRulesets = await toApiRulesets(allServers)
return reply.send({ return reply.send({
rulesets: dbRulesets, rulesets: dbRulesets,
@@ -506,7 +503,7 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
const payload = body.rulesets ?? [] const payload = body.rulesets ?? []
const serverIds = payload.map(r => Number.parseInt(r.serverId, 10)).filter(Number.isFinite) const serverIds = payload.map(r => Number.parseInt(r.serverId, 10)).filter(Number.isFinite)
if (serverIds.length > 0) { if (serverIds.length > 0) {
db.delete(filterRules).where(inArray(filterRules.serverId, serverIds)).run() await db.delete(filterRules).where(inArray(filterRules.serverId, serverIds))
} }
for (const rs of payload) { for (const rs of payload) {
const sid = Number.parseInt(rs.serverId, 10) const sid = Number.parseInt(rs.serverId, 10)
@@ -524,12 +521,12 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "serverId is required" }) return reply.status(400).send({ error: "serverId is required" })
} }
const server = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const server = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
try { try {
app.log.info({ serverId, host: server.host }, "Filters sync from router started") app.log.info({ serverId, host: server.host }, "Filters sync from router started")
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "filters.sync.from_router.started", eventType: "filters.sync.from_router.started",
sourceModule: "filters", sourceModule: "filters",
@@ -541,7 +538,7 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
const remote = await fetchServerFilters(server) const remote = await fetchServerFilters(server)
await replaceDbRules(server.id, remote.rules) await replaceDbRules(server.id, remote.rules)
app.log.info({ serverId, totalRules: remote.rules.length }, "Filters sync from router completed") app.log.info({ serverId, totalRules: remote.rules.length }, "Filters sync from router completed")
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "filters.sync.from_router.done", eventType: "filters.sync.from_router.done",
sourceModule: "filters", sourceModule: "filters",
@@ -553,7 +550,7 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.send({ ok: true, updatedServers: 1, totalRules: remote.rules.length, serverId }) return reply.send({ ok: true, updatedServers: 1, totalRules: remote.rules.length, serverId })
} catch (err) { } catch (err) {
app.log.error({ serverId, err: String(err) }, "Filters sync from router failed") app.log.error({ serverId, err: String(err) }, "Filters sync from router failed")
appendEvent({ await appendEvent({
level: "critical", level: "critical",
eventType: "filters.sync.from_router.failed", eventType: "filters.sync.from_router.failed",
sourceModule: "filters", sourceModule: "filters",
@@ -570,7 +567,7 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
const body = req.body as { serverId?: string | number } | undefined const body = req.body as { serverId?: string | number } | undefined
const requestedServerId = parseDbServerId(body?.serverId) const requestedServerId = parseDbServerId(body?.serverId)
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const targetServers = requestedServerId !== null const targetServers = requestedServerId !== null
? allServers.filter(s => s.id === requestedServerId) ? allServers.filter(s => s.id === requestedServerId)
: allServers : allServers
@@ -582,7 +579,7 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
let updatedServers = 0 let updatedServers = 0
let pushedRules = 0 let pushedRules = 0
const errors: Array<{ serverId: number; error: string }> = [] const errors: Array<{ serverId: number; error: string }> = []
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "filters.sync.to_router.started", eventType: "filters.sync.to_router.started",
sourceModule: "filters", sourceModule: "filters",
@@ -618,10 +615,9 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
.map(r => r[".id"]) .map(r => r[".id"])
.filter((id): id is string => Boolean(id)) .filter((id): id is string => Boolean(id))
const rows = db.select().from(filterRules) const rows = await db.select().from(filterRules)
.where(and(eq(filterRules.serverId, server.id))) .where(and(eq(filterRules.serverId, server.id)))
.orderBy(asc(filterRules.sortOrder)) .orderBy(asc(filterRules.sortOrder))
.all()
const rules: ApiFilterRule[] = rows.map(r => ({ const rules: ApiFilterRule[] = rows.map(r => ({
id: String(r.id), id: String(r.id),
@@ -639,7 +635,7 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
// трактует как «вызов команды» и отдаёт 400 «no such command». // трактует как «вызов команды» и отдаёт 400 «no such command».
// См. https://help.mikrotik.com/docs/spaces/ROS/pages/47579162/REST+API // См. https://help.mikrotik.com/docs/spaces/ROS/pages/47579162/REST+API
if (rules.length > 0) { if (rules.length > 0) {
const ruleBody = toRouterRuleBody(server.id, rules) const ruleBody = await toRouterRuleBody(server.id, rules)
if (managedRule && managedRule[".id"]) { if (managedRule && managedRule[".id"]) {
await client.patch( await client.patch(
`/routing/filter/rule/${encodeURIComponent(managedRule[".id"])}`, `/routing/filter/rule/${encodeURIComponent(managedRule[".id"])}`,
@@ -688,7 +684,7 @@ const filtersRoutes: FastifyPluginAsyncZod = async (app) => {
} }
} }
appendEvent({ await appendEvent({
level: errors.length === 0 ? "info" : "warning", level: errors.length === 0 ? "info" : "warning",
eventType: errors.length === 0 ? "filters.sync.to_router.done" : "filters.sync.to_router.partial", eventType: errors.length === 0 ? "filters.sync.to_router.done" : "filters.sync.to_router.partial",
sourceModule: "filters", sourceModule: "filters",
+11 -11
View File
@@ -116,8 +116,8 @@ function rosErr(e: unknown): string {
return String(e) return String(e)
} }
function requireServer(serverId: string) { async function requireServer(serverId: string) {
return getEnabledServerById(serverId) return await getEnabledServerById(serverId)
} }
const firewallRoutes: FastifyPluginAsyncZod = async (app) => { const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
@@ -132,7 +132,7 @@ const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = requireServer(body.serverId) const server = await requireServer(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
const path = firewallRestPath(body.family as FirewallFamily, body.table as FirewallTable) const path = firewallRestPath(body.family as FirewallFamily, body.table as FirewallTable)
@@ -150,7 +150,7 @@ const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = requireServer(body.serverId) const server = await requireServer(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
const path = `${firewallRestPath(body.family as FirewallFamily, body.table as FirewallTable)}/${encodeRosId(body.rosId)}` const path = `${firewallRestPath(body.family as FirewallFamily, body.table as FirewallTable)}/${encodeRosId(body.rosId)}`
@@ -168,7 +168,7 @@ const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = requireServer(body.serverId) const server = await requireServer(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
const path = `${firewallRestPath(body.family, body.table)}/${encodeRosId(body.rosId)}` const path = `${firewallRestPath(body.family, body.table)}/${encodeRosId(body.rosId)}`
@@ -186,7 +186,7 @@ const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = requireServer(body.serverId) const server = await requireServer(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
const path = `${firewallRestPath(body.family, body.table)}/${encodeRosId(body.rosId)}` const path = `${firewallRestPath(body.family, body.table)}/${encodeRosId(body.rosId)}`
@@ -204,7 +204,7 @@ const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = requireServer(body.serverId) const server = await requireServer(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
const path = `${firewallRestPath(body.family, body.table)}/move` const path = `${firewallRestPath(body.family, body.table)}/move`
@@ -225,7 +225,7 @@ const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = requireServer(body.serverId) const server = await requireServer(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
try { try {
@@ -242,7 +242,7 @@ const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = requireServer(body.serverId) const server = await requireServer(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
const path = `${firewallRestPath(body.family, "address-list")}/${encodeRosId(body.rosId)}` const path = `${firewallRestPath(body.family, "address-list")}/${encodeRosId(body.rosId)}`
@@ -260,7 +260,7 @@ const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = requireServer(body.serverId) const server = await requireServer(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
const path = `${firewallRestPath(body.family, "address-list")}/${encodeRosId(body.rosId)}` const path = `${firewallRestPath(body.family, "address-list")}/${encodeRosId(body.rosId)}`
@@ -278,7 +278,7 @@ const firewallRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = requireServer(body.serverId) const server = await requireServer(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
const path = `${firewallRestPath(body.family, "address-list")}/${encodeRosId(body.rosId)}` const path = `${firewallRestPath(body.family, "address-list")}/${encodeRosId(body.rosId)}`
+11 -9
View File
@@ -8,7 +8,7 @@ import {
const internetPathRoutes: FastifyPluginAsyncZod = async (app) => { const internetPathRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/internet-path/settings", async (_req, reply) => { app.get("/internet-path/settings", async (_req, reply) => {
const s = getInternetPathSettings() const s = await getInternetPathSettings()
return reply.send({ return reply.send({
enabled: s.enabled, enabled: s.enabled,
intervalSec: s.intervalSec, intervalSec: s.intervalSec,
@@ -25,12 +25,12 @@ const internetPathRoutes: FastifyPluginAsyncZod = async (app) => {
intervalSec?: number | string intervalSec?: number | string
retentionDays?: number | string retentionDays?: number | string
} }
const updated = updateInternetPathSettings({ const updated = await updateInternetPathSettings({
enabled: body.enabled, enabled: body.enabled,
intervalSec: body.intervalSec == null ? undefined : Math.max(30, Number.parseInt(String(body.intervalSec), 10) || 300), intervalSec: body.intervalSec == null ? undefined : Math.max(30, Number.parseInt(String(body.intervalSec), 10) || 300),
retentionDays: body.retentionDays == null ? undefined : Math.max(1, Number.parseInt(String(body.retentionDays), 10) || 14), retentionDays: body.retentionDays == null ? undefined : Math.max(1, Number.parseInt(String(body.retentionDays), 10) || 14),
}) })
refreshScheduler() await refreshScheduler()
return reply.send({ return reply.send({
ok: true, ok: true,
settings: { settings: {
@@ -45,13 +45,15 @@ const internetPathRoutes: FastifyPluginAsyncZod = async (app) => {
}) })
app.get("/internet-path/latest", async (_req, reply) => { app.get("/internet-path/latest", async (_req, reply) => {
const row = getLatestInternetPathSnapshot() const row = await getLatestInternetPathSnapshot()
if (!row) return reply.send({ snapshot: null }) if (!row) return reply.send({ snapshot: null })
let payload: unknown = null let payload: unknown = row.payloadJson
try { if (typeof row.payloadJson === "string") {
payload = JSON.parse(row.payloadJson) try {
} catch { payload = JSON.parse(row.payloadJson)
payload = null } catch {
payload = null
}
} }
return reply.send({ return reply.send({
snapshot: payload, snapshot: payload,
+23 -28
View File
@@ -207,23 +207,21 @@ function calcRouteScore(pingMs: number, dlMbps: number, ulMbps: number, pingWeig
return Math.round(w * pingScore + (1 - w) * speedScore) return Math.round(w * pingScore + (1 - w) * speedScore)
} }
function getLatestSnapshotLatencyMs(serverId: number): number { async function getLatestSnapshotLatencyMs(serverId: number): Promise<number> {
const latest = db const latest = await db
.select() .select()
.from(serverSnapshots) .from(serverSnapshots)
.where(eq(serverSnapshots.serverId, serverId)) .where(eq(serverSnapshots.serverId, serverId))
.orderBy(desc(serverSnapshots.polledAt)) .orderBy(desc(serverSnapshots.polledAt))
.limit(1) .limit(1)
.all()
return latest.length > 0 ? Math.max(1, Math.round(latest[0].latencyMs ?? 100)) : 100 return latest.length > 0 ? Math.max(1, Math.round(latest[0].latencyMs ?? 100)) : 100
} }
function latestTrafficByInterface(serverId: number): Map<string, TrafficSampleRow> { async function latestTrafficByInterface(serverId: number): Promise<Map<string, { id: number; serverId: number; disabled: boolean; sampledAt: string; interfaceName: string; peerPublicKey: string; rxBytes: number; txBytes: number; rxBps: number; txBps: number; running: boolean; }>> {
const rows = db const rows = await db
.select() .select()
.from(trafficSamples) .from(trafficSamples)
.where(eq(trafficSamples.serverId, serverId)) .where(eq(trafficSamples.serverId, serverId))
.all()
const map = new Map<string, TrafficSampleRow>() const map = new Map<string, TrafficSampleRow>()
for (const row of rows) { for (const row of rows) {
const prev = map.get(row.interfaceName) const prev = map.get(row.interfaceName)
@@ -302,14 +300,14 @@ async function buildOspfOptimizationPlan(server: ServerRow, pingWeight: number):
const areaMap = buildAreaMap(areas) const areaMap = buildAreaMap(areas)
const parsed = parseInterfaces(server, ifaceTemplates, areas, instances, areaMap) const parsed = parseInterfaces(server, ifaceTemplates, areas, instances, areaMap)
const editable = parsed.filter((i) => !i.disabled && !isRefInterfaceName(i.interface)) const editable = parsed.filter((i) => !i.disabled && !isRefInterfaceName(i.interface))
const localLatencyMs = getLatestSnapshotLatencyMs(server.id) const localLatencyMs = await getLatestSnapshotLatencyMs(server.id)
const latestTraffic = latestTrafficByInterface(server.id) const latestTraffic = await latestTrafficByInterface(server.id)
// 1) Сопоставляем remote OSPF router-id -> сервер из каталога (чтобы взять унифицированный ping как в карте /route-optimizer). // 1) Сопоставляем remote OSPF router-id -> сервер из каталога (чтобы взять унифицированный ping как в карте /route-optimizer).
const neededRouterIds = new Set(neighbors.map((n) => String(n["router-id"] ?? "")).filter((x) => x.length > 0)) const neededRouterIds = new Set(neighbors.map((n) => String(n["router-id"] ?? "")).filter((x) => x.length > 0))
async function buildRouterIdToServerMap(needed: Set<string>): Promise<Map<string, ServerRow["id"]>> { async function buildRouterIdToServerMap(needed: Set<string>): Promise<Map<string, ServerRow["id"]>> {
const enabledServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const enabledServers = await db.select().from(servers).where(eq(servers.enabled, true))
const out = new Map<string, ServerRow["id"]>() const out = new Map<string, ServerRow["id"]>()
const neededLeft = new Set(needed) const neededLeft = new Set(needed)
@@ -356,11 +354,10 @@ async function buildOspfOptimizationPlan(server: ServerRow, pingWeight: number):
} }
// 3) Предгружаем uptimeSpeedProbes: оттуда берём и ping, и скорость (единый источник как у карты). // 3) Предгружаем uptimeSpeedProbes: оттуда берём и ping, и скорость (единый источник как у карты).
const allSourceProbes = db const allSourceProbes = (await db
.select() .select()
.from(uptimeSpeedProbes) .from(uptimeSpeedProbes)
.where(eq(uptimeSpeedProbes.srcServerId, server.id)) .where(eq(uptimeSpeedProbes.srcServerId, server.id)))
.all()
.filter((r: SpeedProbeRow) => r.enabled !== false) .filter((r: SpeedProbeRow) => r.enabled !== false)
const speedProbesByDest = new Map<number, SpeedProbeRow[]>() const speedProbesByDest = new Map<number, SpeedProbeRow[]>()
for (const p of allSourceProbes) { for (const p of allSourceProbes) {
@@ -482,8 +479,7 @@ async function buildOspfOptimizationPlan(server: ServerRow, pingWeight: number):
for (const [iface, probe] of assigned.entries()) assignedProbeByIface.set(iface, probe) for (const [iface, probe] of assigned.entries()) assignedProbeByIface.set(iface, probe)
} }
return editable return (await Promise.all(editable.map(async (iface) => {
.map((iface) => {
const t = latestTraffic.get(iface.interface) const t = latestTraffic.get(iface.interface)
const speed = normalizeSpeedMbps(t) const speed = normalizeSpeedMbps(t)
@@ -494,7 +490,7 @@ async function buildOspfOptimizationPlan(server: ServerRow, pingWeight: number):
bestProbe?.lastPingRttMs != null bestProbe?.lastPingRttMs != null
? Math.max(1, Math.round(bestProbe.lastPingRttMs)) ? Math.max(1, Math.round(bestProbe.lastPingRttMs))
: dstServerId != null : dstServerId != null
? Math.min(995, Math.round(localLatencyMs + getLatestSnapshotLatencyMs(dstServerId))) ? Math.min(995, Math.round(localLatencyMs + await getLatestSnapshotLatencyMs(dstServerId)))
: localLatencyMs : localLatencyMs
const dlMbps = const dlMbps =
@@ -517,8 +513,7 @@ async function buildOspfOptimizationPlan(server: ServerRow, pingWeight: number):
score: calcRouteScore(pingMs, dlMbps, ulMbps, pingWeight), score: calcRouteScore(pingMs, dlMbps, ulMbps, pingWeight),
optimalCost: 0, optimalCost: 0,
} }
}) }))).sort((a, b) => b.score - a.score || a.interface.localeCompare(b.interface))
.sort((a, b) => b.score - a.score || a.interface.localeCompare(b.interface))
.map((row, idx) => ({ ...row, optimalCost: (idx + 1) * 10 })) .map((row, idx) => ({ ...row, optimalCost: (idx + 1) * 10 }))
} }
@@ -528,7 +523,7 @@ const ospfRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/ospf/neighbors — aggregate OSPF neighbors from ALL enabled servers // GET /api/ospf/neighbors — aggregate OSPF neighbors from ALL enabled servers
app.get("/ospf/neighbors", async (_req, reply) => { app.get("/ospf/neighbors", async (_req, reply) => {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const results = await Promise.all( const results = await Promise.all(
allServers.map(async (server) => { allServers.map(async (server) => {
@@ -547,7 +542,7 @@ const ospfRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/ospf/interfaces — aggregate OSPF interface templates from ALL enabled servers // GET /api/ospf/interfaces — aggregate OSPF interface templates from ALL enabled servers
app.get("/ospf/interfaces", async (_req, reply) => { app.get("/ospf/interfaces", async (_req, reply) => {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const results = await Promise.all( const results = await Promise.all(
allServers.map(async (server) => { allServers.map(async (server) => {
@@ -566,7 +561,7 @@ const ospfRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/ospf/instances — aggregate OSPF instances from ALL enabled servers // GET /api/ospf/instances — aggregate OSPF instances from ALL enabled servers
app.get("/ospf/instances", async (_req, reply) => { app.get("/ospf/instances", async (_req, reply) => {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const results = await Promise.all( const results = await Promise.all(
allServers.map(async (server) => { allServers.map(async (server) => {
@@ -584,7 +579,7 @@ const ospfRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/ospf/all — single round-trip: neighbors + interfaces + instances + BFD // GET /api/ospf/all — single round-trip: neighbors + interfaces + instances + BFD
app.get("/ospf/all", async (_req, reply) => { app.get("/ospf/all", async (_req, reply) => {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const perServer = await Promise.all( const perServer = await Promise.all(
allServers.map(async (server) => { allServers.map(async (server) => {
@@ -613,7 +608,7 @@ const ospfRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/bfd/sessions — BFD sessions only (for direct access) // GET /api/bfd/sessions — BFD sessions only (for direct access)
app.get("/bfd/sessions", async (_req, reply) => { app.get("/bfd/sessions", async (_req, reply) => {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const results = await Promise.all( const results = await Promise.all(
allServers.map(async (server) => { allServers.map(async (server) => {
try { try {
@@ -631,10 +626,10 @@ const ospfRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/servers/:id/ospf — single server OSPF + BFD data // GET /api/servers/:id/ospf — single server OSPF + BFD data
app.get("/servers/:id/ospf", { schema: { params: ServerIdParamSchema } }, async (req, reply) => { app.get("/servers/:id/ospf", { schema: { params: ServerIdParamSchema } }, async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const server = db const server = (await db
.select().from(servers) .select().from(servers)
.where(eq(servers.id, params.id)) .where(eq(servers.id, params.id))
.limit(1).all()[0] .limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
@@ -659,10 +654,10 @@ const ospfRoutes: FastifyPluginAsyncZod = async (app) => {
{ schema: { params: ServerIdParamSchema, body: OspfOptimizeBodySchema } }, { schema: { params: ServerIdParamSchema, body: OspfOptimizeBodySchema } },
async (req, reply) => { async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const server = db const server = (await db
.select().from(servers) .select().from(servers)
.where(eq(servers.id, params.id)) .where(eq(servers.id, params.id))
.limit(1).all()[0] .limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
@@ -691,10 +686,10 @@ const ospfRoutes: FastifyPluginAsyncZod = async (app) => {
{ schema: { params: ServerIdParamSchema, body: OspfOptimizeBodySchema } }, { schema: { params: ServerIdParamSchema, body: OspfOptimizeBodySchema } },
async (req, reply) => { async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const server = db const server = (await db
.select().from(servers) .select().from(servers)
.where(eq(servers.id, params.id)) .where(eq(servers.id, params.id))
.limit(1).all()[0] .limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
+6 -6
View File
@@ -256,14 +256,14 @@ async function mtuDiscover(client: MikrotikClient, address: string, srcIpv4: str
return lines.join("\n") return lines.join("\n")
} }
function resolveBtestPeer(remoteHost: string, explicitDstId?: number) { async function resolveBtestPeer(remoteHost: string, explicitDstId?: number) {
if (explicitDstId != null && Number.isFinite(explicitDstId)) { if (explicitDstId != null && Number.isFinite(explicitDstId)) {
const s = db.select().from(servers).where(eq(servers.id, explicitDstId)).limit(1).all()[0] const s = (await db.select().from(servers).where(eq(servers.id, explicitDstId)).limit(1))[0]
if (s) return s if (s) return s
} }
const norm = remoteHost.trim().toLowerCase() const norm = remoteHost.trim().toLowerCase()
return db.select().from(servers).where(eq(servers.enabled, true)).all() const enabled = await db.select().from(servers).where(eq(servers.enabled, true))
.find((x) => x.host.trim().toLowerCase() === norm) return enabled.find((x) => x.host.trim().toLowerCase() === norm)
} }
const probesRoutes: FastifyPluginAsyncZod = async (app) => { const probesRoutes: FastifyPluginAsyncZod = async (app) => {
@@ -271,7 +271,7 @@ const probesRoutes: FastifyPluginAsyncZod = async (app) => {
const sid = parseServerId(String((req.params as { id?: string }).id ?? "")) const sid = parseServerId(String((req.params as { id?: string }).id ?? ""))
if (sid === null) return reply.status(400).send({ error: "Invalid server id" }) if (sid === null) return reply.status(400).send({ error: "Invalid server id" })
const server = db.select().from(servers).where(eq(servers.id, sid)).limit(1).all()[0] const server = (await db.select().from(servers).where(eq(servers.id, sid)).limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
const body = req.body as { const body = req.body as {
@@ -369,7 +369,7 @@ const probesRoutes: FastifyPluginAsyncZod = async (app) => {
case "bandwidth": { case "bandwidth": {
const remote = String(body.bwRemoteAddress ?? "").trim() const remote = String(body.bwRemoteAddress ?? "").trim()
if (!remote) return reply.status(400).send({ error: "bwRemoteAddress required" }) if (!remote) return reply.status(400).send({ error: "bwRemoteAddress required" })
const dst = resolveBtestPeer(remote, body.dstServerId) const dst = await resolveBtestPeer(remote, body.dstServerId)
if (!dst) { if (!dst) {
return reply.status(400).send({ return reply.status(400).send({
error: "Не найден сервер назначения для bandwidth-test: добавьте узел с host = GRE remote или укажите dstServerId", error: "Не найден сервер назначения для bandwidth-test: добавьте узел с host = GRE remote или укажите dstServerId",
+13 -14
View File
@@ -80,13 +80,12 @@ function splitGateway(raw: string): { ip: string; name: string } | null {
return { ip, name: name || ip } return { ip, name: name || ip }
} }
function mapDbRoutes(serverId: number): RecursiveRouteDto[] { async function mapDbRoutes(serverId: number): Promise<RecursiveRouteDto[]> {
const rows = db const rows = await db
.select() .select()
.from(recursiveRoutes) .from(recursiveRoutes)
.where(eq(recursiveRoutes.serverId, serverId)) .where(eq(recursiveRoutes.serverId, serverId))
.orderBy(asc(recursiveRoutes.sortOrder)) .orderBy(asc(recursiveRoutes.sortOrder))
.all()
return rows.map((r) => ({ return rows.map((r) => ({
id: String(r.id), id: String(r.id),
@@ -118,10 +117,10 @@ function toRouterPayload(route: RecursiveRouteDto): Record<string, string> {
} }
async function replaceDbRoutes(serverId: number, routes: RecursiveRouteDto[]) { async function replaceDbRoutes(serverId: number, routes: RecursiveRouteDto[]) {
db.delete(recursiveRoutes).where(eq(recursiveRoutes.serverId, serverId)).run() await db.delete(recursiveRoutes).where(eq(recursiveRoutes.serverId, serverId))
if (routes.length === 0) return if (routes.length === 0) return
const now = new Date().toISOString() const now = new Date().toISOString()
db.insert(recursiveRoutes).values( await db.insert(recursiveRoutes).values(
routes.map((r, i) => ({ routes.map((r, i) => ({
serverId, serverId,
sortOrder: i, sortOrder: i,
@@ -138,7 +137,7 @@ async function replaceDbRoutes(serverId: number, routes: RecursiveRouteDto[]) {
createdAt: now, createdAt: now,
updatedAt: now, updatedAt: now,
})), })),
).run() )
} }
const recursiveRoutesPlugin: FastifyPluginAsyncZod = async (app) => { const recursiveRoutesPlugin: FastifyPluginAsyncZod = async (app) => {
@@ -146,7 +145,7 @@ const recursiveRoutesPlugin: FastifyPluginAsyncZod = async (app) => {
const q = req.query as { serverId?: string | number } const q = req.query as { serverId?: string | number }
const serverId = parseDbServerId(q.serverId) const serverId = parseDbServerId(q.serverId)
if (serverId === null) return reply.status(400).send({ error: "serverId is required" }) if (serverId === null) return reply.status(400).send({ error: "serverId is required" })
const server = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const server = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
try { try {
@@ -176,16 +175,16 @@ const recursiveRoutesPlugin: FastifyPluginAsyncZod = async (app) => {
const q = req.query as { serverId?: string | number } const q = req.query as { serverId?: string | number }
const serverId = parseDbServerId(q.serverId) const serverId = parseDbServerId(q.serverId)
if (serverId === null) return reply.status(400).send({ error: "serverId is required" }) if (serverId === null) return reply.status(400).send({ error: "serverId is required" })
const server = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const server = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
return reply.send({ routes: mapDbRoutes(serverId) }) return reply.send({ routes: await mapDbRoutes(serverId) })
}) })
app.put("/recursive-routes", async (req, reply) => { app.put("/recursive-routes", async (req, reply) => {
const body = req.body as { serverId?: string | number; routes?: RecursiveRouteDto[] } const body = req.body as { serverId?: string | number; routes?: RecursiveRouteDto[] }
const serverId = parseDbServerId(body.serverId) const serverId = parseDbServerId(body.serverId)
if (serverId === null) return reply.status(400).send({ error: "serverId is required" }) if (serverId === null) return reply.status(400).send({ error: "serverId is required" })
const server = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const server = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
await replaceDbRoutes(serverId, body.routes ?? []) await replaceDbRoutes(serverId, body.routes ?? [])
return reply.send({ ok: true }) return reply.send({ ok: true })
@@ -195,10 +194,10 @@ const recursiveRoutesPlugin: FastifyPluginAsyncZod = async (app) => {
const body = req.body as { serverId?: string | number } | undefined const body = req.body as { serverId?: string | number } | undefined
const serverId = parseDbServerId(body?.serverId) const serverId = parseDbServerId(body?.serverId)
if (serverId === null) return reply.status(400).send({ error: "serverId is required" }) if (serverId === null) return reply.status(400).send({ error: "serverId is required" })
const server: ServerRow | undefined = db const server: ServerRow | undefined = (await db
.select().from(servers) .select().from(servers)
.where(eq(servers.id, serverId)) .where(eq(servers.id, serverId))
.limit(1).all()[0] .limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
try { try {
@@ -231,7 +230,7 @@ const recursiveRoutesPlugin: FastifyPluginAsyncZod = async (app) => {
const body = req.body as { serverId?: string | number } | undefined const body = req.body as { serverId?: string | number } | undefined
const serverId = parseDbServerId(body?.serverId) const serverId = parseDbServerId(body?.serverId)
if (serverId === null) return reply.status(400).send({ error: "serverId is required" }) if (serverId === null) return reply.status(400).send({ error: "serverId is required" })
const server = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const server = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
try { try {
@@ -243,7 +242,7 @@ const recursiveRoutesPlugin: FastifyPluginAsyncZod = async (app) => {
await client.delete(`/ip/route/${encodeURIComponent(r[".id"])}`) await client.delete(`/ip/route/${encodeURIComponent(r[".id"])}`)
} }
const dbRows = mapDbRoutes(serverId) const dbRows = await mapDbRoutes(serverId)
for (const route of dbRows) { for (const route of dbRows) {
await client.post("/ip/route", toRouterPayload(route)) await client.post("/ip/route", toRouterPayload(route))
} }
+5 -5
View File
@@ -14,7 +14,7 @@ function parseJobKey(raw: string | undefined): typeof JOB_KEYS[number] | null {
const schedulerRoutes: FastifyPluginAsyncZod = async (app) => { const schedulerRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/scheduler/status", async (_req, reply) => { app.get("/scheduler/status", async (_req, reply) => {
return reply.send(getSchedulerStatus()) return reply.send(await getSchedulerStatus())
}) })
app.get("/scheduler/runs", async (req, reply) => { app.get("/scheduler/runs", async (req, reply) => {
@@ -25,12 +25,12 @@ const schedulerRoutes: FastifyPluginAsyncZod = async (app) => {
} }
const limit = Math.min(200, Math.max(1, Number.parseInt(String(q.limit ?? "50"), 10) || 50)) const limit = Math.min(200, Math.max(1, Number.parseInt(String(q.limit ?? "50"), 10) || 50))
const offset = Math.max(0, Number.parseInt(String(q.offset ?? "0"), 10) || 0) const offset = Math.max(0, Number.parseInt(String(q.offset ?? "0"), 10) || 0)
return reply.send(listSchedulerRuns({ jobKey: jobKey ?? undefined, limit, offset })) return reply.send(await listSchedulerRuns({ jobKey: jobKey ?? undefined, limit, offset }))
}) })
app.post("/scheduler/refresh", async (_req, reply) => { app.post("/scheduler/refresh", async (_req, reply) => {
refreshScheduler() await refreshScheduler()
return reply.send({ ok: true, status: getSchedulerStatus() }) return reply.send({ ok: true, status: await getSchedulerStatus() })
}) })
app.post("/scheduler/jobs/:jobKey/run-now", async (req, reply) => { app.post("/scheduler/jobs/:jobKey/run-now", async (req, reply) => {
@@ -38,7 +38,7 @@ const schedulerRoutes: FastifyPluginAsyncZod = async (app) => {
if (!jobKey) return reply.status(400).send({ error: "Некорректный jobKey" }) if (!jobKey) return reply.status(400).send({ error: "Некорректный jobKey" })
try { try {
await runSchedulerJobNow(jobKey) await runSchedulerJobNow(jobKey)
return reply.send({ ok: true, status: getSchedulerStatus() }) return reply.send({ ok: true, status: await getSchedulerStatus() })
} catch (e) { } catch (e) {
const statusCode = (e as Error & { statusCode?: number }).statusCode const statusCode = (e as Error & { statusCode?: number }).statusCode
if (statusCode === 409) { if (statusCode === 409) {
+4 -4
View File
@@ -10,7 +10,7 @@ import {
const serversApiPingRoutes: FastifyPluginAsyncZod = async (app) => { const serversApiPingRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/servers-api-ping/settings", async (_req, reply) => { app.get("/servers-api-ping/settings", async (_req, reply) => {
const settings = getServersApiPingSettings() const settings = await getServersApiPingSettings()
const state = getServersRestPingCollectorState() const state = getServersRestPingCollectorState()
return reply.send({ return reply.send({
enabled: settings.enabled, enabled: settings.enabled,
@@ -29,11 +29,11 @@ const serversApiPingRoutes: FastifyPluginAsyncZod = async (app) => {
} }
const intervalSec = const intervalSec =
body.intervalSec == null ? undefined : Math.max(10, Number.parseInt(String(body.intervalSec), 10) || 120) body.intervalSec == null ? undefined : Math.max(10, Number.parseInt(String(body.intervalSec), 10) || 120)
const updated = updateServersApiPingSettings({ const updated = await updateServersApiPingSettings({
enabled: body.enabled, enabled: body.enabled,
intervalSec, intervalSec,
}) })
refreshScheduler() await refreshScheduler()
return reply.send({ return reply.send({
ok: true, ok: true,
settings: { settings: {
@@ -49,7 +49,7 @@ const serversApiPingRoutes: FastifyPluginAsyncZod = async (app) => {
app.post("/servers-api-ping/collect-now", async (_req, reply) => { app.post("/servers-api-ping/collect-now", async (_req, reply) => {
await collectServersRestPingOnce() await collectServersRestPingOnce()
scheduleAlertEngineAfterDataCollectors() scheduleAlertEngineAfterDataCollectors()
const row = getServersApiPingSettings() const row = await getServersApiPingSettings()
return reply.send({ return reply.send({
ok: true, ok: true,
lastCollectedAt: row.lastCollectedAt ?? null, lastCollectedAt: row.lastCollectedAt ?? null,
+15 -13
View File
@@ -77,13 +77,13 @@ const serversRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/servers // GET /api/servers
app.get("/", async (_req, reply) => { app.get("/", async (_req, reply) => {
return reply.send(listServersRead()) return reply.send(await listServersRead())
}) })
// GET /api/servers/:id/ros-src-address — IPv4 для src-address в RouterOS (не FQDN) // GET /api/servers/:id/ros-src-address — IPv4 для src-address в RouterOS (не FQDN)
app.get("/:id/ros-src-address", { schema: { params: ServerIdParamSchema } }, async (req, reply) => { app.get("/:id/ros-src-address", { schema: { params: ServerIdParamSchema } }, async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const server = getServerRowById(params.id) const server = await getServerRowById(params.id)
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
const ipv4 = await resolveRosSrcIpv4(server.host) const ipv4 = await resolveRosSrcIpv4(server.host)
@@ -93,7 +93,7 @@ const serversRoutes: FastifyPluginAsyncZod = async (app) => {
// GET /api/servers/:id/wan-runtime — DHCP lease + active default route for HomeRouter WAN uplinks // GET /api/servers/:id/wan-runtime — DHCP lease + active default route for HomeRouter WAN uplinks
app.get("/:id/wan-runtime", { schema: { params: ServerIdParamSchema } }, async (req, reply) => { app.get("/:id/wan-runtime", { schema: { params: ServerIdParamSchema } }, async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const server = getServerReadById(params.id) const server = await getServerReadById(params.id)
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
const toIp = (raw: string | null | undefined): string | null => { const toIp = (raw: string | null | undefined): string | null => {
@@ -141,7 +141,9 @@ const serversRoutes: FastifyPluginAsyncZod = async (app) => {
} }
try { try {
const client = MikrotikClient.fromServer(getServerRowById(params.id)!) const row = await getServerRowById(params.id)
if (!row) return reply.status(404).send({ error: "Server not found" })
const client = MikrotikClient.fromServer(row)
const isTrue = (v: unknown) => { const isTrue = (v: unknown) => {
const s = String(v ?? "").trim().toLowerCase() const s = String(v ?? "").trim().toLowerCase()
return s === "true" || s === "yes" return s === "true" || s === "yes"
@@ -228,13 +230,13 @@ const serversRoutes: FastifyPluginAsyncZod = async (app) => {
// POST /api/servers // POST /api/servers
app.post("/", { schema: { body: ServerCreateSchema } }, async (req, reply) => { app.post("/", { schema: { body: ServerCreateSchema } }, async (req, reply) => {
return reply.status(201).send(createServer(req.body as ServerCreateRequest)) return reply.status(201).send(await createServer(req.body as ServerCreateRequest))
}) })
// GET /api/servers/:id // GET /api/servers/:id
app.get("/:id", { schema: { params: ServerIdParamSchema } }, async (req, reply) => { app.get("/:id", { schema: { params: ServerIdParamSchema } }, async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const server = getServerReadById(params.id) const server = await getServerReadById(params.id)
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
return reply.send(server) return reply.send(server)
}) })
@@ -245,25 +247,25 @@ const serversRoutes: FastifyPluginAsyncZod = async (app) => {
{ schema: { params: ServerIdParamSchema, body: ServerUpdateSchema } }, { schema: { params: ServerIdParamSchema, body: ServerUpdateSchema } },
async (req, reply) => { async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const existing = getServerReadById(params.id) const existing = await getServerReadById(params.id)
if (!existing) return reply.status(404).send({ error: "Server not found" }) if (!existing) return reply.status(404).send({ error: "Server not found" })
return reply.send(updateServer(params.id, req.body as ServerUpdateRequest)) return reply.send(await updateServer(params.id, req.body as ServerUpdateRequest))
}, },
) )
// DELETE /api/servers/:id // DELETE /api/servers/:id
app.delete("/:id", { schema: { params: ServerIdParamSchema } }, async (req, reply) => { app.delete("/:id", { schema: { params: ServerIdParamSchema } }, async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const existing = getServerReadById(params.id) const existing = await getServerReadById(params.id)
if (!existing) return reply.status(404).send({ error: "Server not found" }) if (!existing) return reply.status(404).send({ error: "Server not found" })
deleteServer(params.id) await deleteServer(params.id)
return reply.status(204).send() return reply.status(204).send()
}) })
// POST /api/servers/:id/poll // POST /api/servers/:id/poll
app.post("/:id/poll", { schema: { params: ServerIdParamSchema } }, async (req, reply) => { app.post("/:id/poll", { schema: { params: ServerIdParamSchema } }, async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const existing = getServerReadById(params.id) const existing = await getServerReadById(params.id)
if (!existing) return reply.status(404).send({ error: "Server not found" }) if (!existing) return reply.status(404).send({ error: "Server not found" })
try { try {
@@ -281,9 +283,9 @@ const serversRoutes: FastifyPluginAsyncZod = async (app) => {
async (req, reply) => { async (req, reply) => {
const params = req.params as ServerIdParams const params = req.params as ServerIdParams
const query = req.query as SnapshotsQuery const query = req.query as SnapshotsQuery
const existing = getServerReadById(params.id) const existing = await getServerReadById(params.id)
if (!existing) return reply.status(404).send({ error: "Server not found" }) if (!existing) return reply.status(404).send({ error: "Server not found" })
return reply.send(listServerSnapshots(params.id, query.limit)) return reply.send(await listServerSnapshots(params.id, query.limit))
}, },
) )
} }
+12 -10
View File
@@ -12,22 +12,24 @@ import {
} from "../db/schema.js" } from "../db/schema.js"
import { listUsers } from "../modules/users/service/users-service.js" import { listUsers } from "../modules/users/service/users-service.js"
function tableCount(table: typeof servers | typeof filterRules | typeof uptimeProbes | typeof uptimeSpeedProbes | typeof recursiveRoutes): number { async function tableCount(table: typeof servers | typeof filterRules | typeof uptimeProbes | typeof uptimeSpeedProbes | typeof recursiveRoutes): Promise<number> {
return db.select({ n: count() }).from(table).all()[0]?.n ?? 0 const rows = await db.select({ n: count() }).from(table)
return rows[0]?.n ?? 0
} }
const sidebarCountsRoutes: FastifyPluginAsyncZod = async (app) => { const sidebarCountsRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/sidebar-counts", async (_req, reply) => { app.get("/sidebar-counts", async (_req, reply) => {
const serversTotal = tableCount(servers) const serversTotal = await tableCount(servers)
const filterRulesTotal = tableCount(filterRules) const filterRulesTotal = await tableCount(filterRules)
const uptimeProbesTotal = tableCount(uptimeProbes) const uptimeProbesTotal = await tableCount(uptimeProbes)
const uptimeSpeedProbesTotal = tableCount(uptimeSpeedProbes) const uptimeSpeedProbesTotal = await tableCount(uptimeSpeedProbes)
const recursiveRoutesTotal = tableCount(recursiveRoutes) const recursiveRoutesTotal = await tableCount(recursiveRoutes)
const [certificatesTotal, wireguardTotal] = await Promise.all([ const [certRes, wireguardTotal] = await Promise.all([
listCertificatesFromServers().then((res) => res.certificates.length), listCertificatesFromServers(),
countWireGuardInterfaces().catch(() => 0), countWireGuardInterfaces().catch(() => 0),
]) ])
const usersTotal = listUsers().length const certificatesTotal = certRes.certificates.length
const usersTotal = (await listUsers()).length
return reply.send({ return reply.send({
servers: serversTotal, servers: serversTotal,
+4 -4
View File
@@ -18,7 +18,7 @@ const systemDatabaseRoutes: FastifyPluginAsync = async (app) => {
app.get("/system/database/backup", async (_req, reply) => { app.get("/system/database/backup", async (_req, reply) => {
try { try {
const { filename, buffer } = await exportSystemDatabaseBackup() const { filename, buffer } = await exportSystemDatabaseBackup()
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "system.database.backup", eventType: "system.database.backup",
sourceModule: "system", sourceModule: "system",
@@ -44,16 +44,16 @@ const systemDatabaseRoutes: FastifyPluginAsync = async (app) => {
app.post("/system/database/restore", async (req, reply) => { app.post("/system/database/restore", async (req, reply) => {
const body = req.body const body = req.body
if (!Buffer.isBuffer(body) || body.length === 0) { if (!Buffer.isBuffer(body) || body.length === 0) {
return reply.status(400).send({ error: "Ожидается тело запроса с файлом SQLite" }) return reply.status(400).send({ error: "Ожидается тело запроса с файлом pg_dump (custom, PGDMP)" })
} }
try { try {
await restoreSystemDatabaseBackup(body) await restoreSystemDatabaseBackup(body)
appendEvent({ await appendEvent({
level: "warning", level: "warning",
eventType: "system.database.restore", eventType: "system.database.restore",
sourceModule: "system", sourceModule: "system",
title: "Восстановлена база приложения", title: "Восстановлена база приложения",
message: "Данные SQLite заменены из загруженного файла", message: "Данные PostgreSQL заменены из загруженного дампа",
entityType: "system_database", entityType: "system_database",
entityId: "restore", entityId: "restore",
payload: { payload: {
+16 -16
View File
@@ -94,7 +94,7 @@ function analyticsQuery(req: FastifyRequest) {
async function sendFlowTalkers(req: FastifyRequest, reply: FastifyReply) { async function sendFlowTalkers(req: FastifyRequest, reply: FastifyReply) {
const q = req.query as { range?: string } const q = req.query as { range?: string }
return reply.send(listFlowTalkers(rangeToMinutes(q.range))) return reply.send(await listFlowTalkers(rangeToMinutes(q.range)))
} }
function requestPublicHost(req: FastifyRequest): string { function requestPublicHost(req: FastifyRequest): string {
@@ -145,7 +145,7 @@ function sleep(ms: number, signal: AbortSignal): Promise<void> {
const trafficFlowRoutes: FastifyPluginAsyncZod = async (app) => { const trafficFlowRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/traffic/flow/settings", async (_req, reply) => { app.get("/traffic/flow/settings", async (_req, reply) => {
return reply.send(toTrafficFlowSettingsDto(getFlowListenerState())) return reply.send(await toTrafficFlowSettingsDto(getFlowListenerState()))
}) })
app.put("/traffic/flow/settings", async (req, reply) => { app.put("/traffic/flow/settings", async (req, reply) => {
@@ -153,31 +153,31 @@ const trafficFlowRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
updateTrafficFlowSettings(parsed.data) await updateTrafficFlowSettings(parsed.data)
startTrafficFlowListener() await startTrafficFlowListener()
return reply.send({ ok: true, settings: toTrafficFlowSettingsDto(getFlowListenerState()) }) return reply.send({ ok: true, settings: await toTrafficFlowSettingsDto(getFlowListenerState()) })
}) })
app.post("/traffic/flow/settings/generate-keys", async (_req, reply) => { app.post("/traffic/flow/settings/generate-keys", async (_req, reply) => {
const result = ensureHostKeys() const result = await ensureHostKeys()
return reply.send({ return reply.send({
ok: true, ok: true,
created: result.created, created: result.created,
publicKey: result.publicKey, publicKey: result.publicKey,
settings: toTrafficFlowSettingsDto(getFlowListenerState()), settings: await toTrafficFlowSettingsDto(getFlowListenerState()),
}) })
}) })
app.get("/traffic/flow/host-files", async (_req, reply) => { app.get("/traffic/flow/host-files", async (_req, reply) => {
const row = getTrafficFlowSettingsRow() const row = await getTrafficFlowSettingsRow()
if (!row.hostPrivateKey) ensureHostKeys() if (!row.hostPrivateKey) await ensureHostKeys()
return reply.send({ files: listTrafficFlowHostFiles() }) return reply.send({ files: await listTrafficFlowHostFiles() })
}) })
app.post("/traffic/flow/purge", async (_req, reply) => { app.post("/traffic/flow/purge", async (_req, reply) => {
try { try {
const result = await purgeTrafficFlowStore() const result = await purgeTrafficFlowStore()
appendEvent({ await appendEvent({
level: "warning", level: "warning",
eventType: "traffic.flow.purge", eventType: "traffic.flow.purge",
sourceModule: "traffic", sourceModule: "traffic",
@@ -211,20 +211,20 @@ const trafficFlowRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/traffic/flow/exporters", async (req, reply) => { app.get("/traffic/flow/exporters", async (req, reply) => {
const q = req.query as { range?: string } const q = req.query as { range?: string }
return reply.send(listFlowExporters(rangeToMinutes(q.range))) return reply.send(await listFlowExporters(rangeToMinutes(q.range)))
}) })
app.get("/traffic/flow/clients", async (req, reply) => { app.get("/traffic/flow/clients", async (req, reply) => {
const q = req.query as { range?: string } const q = req.query as { range?: string }
return reply.send(listFlowClients(rangeToMinutes(q.range))) return reply.send(await listFlowClients(rangeToMinutes(q.range)))
}) })
app.get("/traffic/flow/analytics", async (req, reply) => { app.get("/traffic/flow/analytics", async (req, reply) => {
return reply.send(buildFlowAnalytics(analyticsQuery(req))) return reply.send(await buildFlowAnalytics(analyticsQuery(req)))
}) })
app.get("/traffic/flow/map-hops", async (req, reply) => { app.get("/traffic/flow/map-hops", async (req, reply) => {
return reply.send(buildFlowMapHops(analyticsQuery(req))) return reply.send(await buildFlowMapHops(analyticsQuery(req)))
}) })
app.get("/traffic/flow/monthly", async (req, reply) => { app.get("/traffic/flow/monthly", async (req, reply) => {
@@ -233,7 +233,7 @@ const trafficFlowRoutes: FastifyPluginAsyncZod = async (app) => {
const month = /^\d{4}-\d{2}$/.test(q.month ?? "") const month = /^\d{4}-\d{2}$/.test(q.month ?? "")
? (q.month as string) ? (q.month as string)
: `${now.getUTCFullYear()}-${String(now.getUTCMonth() + 1).padStart(2, "0")}` : `${now.getUTCFullYear()}-${String(now.getUTCMonth() + 1).padStart(2, "0")}`
return reply.send(getFlowMonthly(month, parseId(q.serverId))) return reply.send(await getFlowMonthly(month, parseId(q.serverId)))
}) })
app.get("/traffic/flow/live", async (req, reply) => { app.get("/traffic/flow/live", async (req, reply) => {
+26 -27
View File
@@ -57,14 +57,13 @@ interface TrafficInterfaceDto {
const LIVE_TICK_MS = 1500 const LIVE_TICK_MS = 1500
const LIVE_ROS_TIMEOUT_MS = 4000 const LIVE_ROS_TIMEOUT_MS = 4000
function latestSnapshot(serverId: number): SnapshotRow | undefined { async function latestSnapshot(serverId: number) {
return db return (await db
.select() .select()
.from(serverSnapshots) .from(serverSnapshots)
.where(eq(serverSnapshots.serverId, serverId)) .where(eq(serverSnapshots.serverId, serverId))
.orderBy(desc(serverSnapshots.polledAt)) .orderBy(desc(serverSnapshots.polledAt))
.limit(1) .limit(1))[0]
.all()[0]
} }
function rangeToMinutes(range: string | undefined): number { function rangeToMinutes(range: string | undefined): number {
@@ -114,8 +113,8 @@ function buildServerTraffic(
} }
} }
function snapshotStatus(serverId: number): TrafficServerDto["status"] { async function snapshotStatus(serverId: number): Promise<TrafficServerDto["status"]> {
const snap = latestSnapshot(serverId) const snap = await latestSnapshot(serverId)
return snap?.status === "offline" ? "offline" : (snap?.status === "online" ? "online" : "degraded") return snap?.status === "offline" ? "offline" : (snap?.status === "online" ? "online" : "degraded")
} }
@@ -222,8 +221,8 @@ async function monitorTrafficOnce(
const trafficRoutes: FastifyPluginAsyncZod = async (app) => { const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/traffic/settings", async (_req, reply) => { app.get("/traffic/settings", async (_req, reply) => {
const settings = getTrafficSettings() const settings = await getTrafficSettings()
const state = getTrafficCollectorState() const state = await getTrafficCollectorState()
return reply.send({ return reply.send({
enabled: settings.enabled, enabled: settings.enabled,
intervalSec: settings.intervalSec, intervalSec: settings.intervalSec,
@@ -243,12 +242,12 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
} }
const intervalSec = body.intervalSec == null ? undefined : Math.max(5, Number.parseInt(String(body.intervalSec), 10) || 30) const intervalSec = body.intervalSec == null ? undefined : Math.max(5, Number.parseInt(String(body.intervalSec), 10) || 30)
const retentionDays = body.retentionDays == null ? undefined : Math.max(1, Number.parseInt(String(body.retentionDays), 10) || 14) const retentionDays = body.retentionDays == null ? undefined : Math.max(1, Number.parseInt(String(body.retentionDays), 10) || 14)
const updated = updateTrafficSettings({ const updated = await updateTrafficSettings({
enabled: body.enabled, enabled: body.enabled,
intervalSec, intervalSec,
retentionDays, retentionDays,
}) })
refreshScheduler() await refreshScheduler()
return reply.send({ return reply.send({
ok: true, ok: true,
settings: { settings: {
@@ -266,8 +265,8 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
try { try {
await collectTrafficOnce() await collectTrafficOnce()
scheduleAlertEngineAfterDataCollectors() scheduleAlertEngineAfterDataCollectors()
const updated = getTrafficSettings() const updated = await getTrafficSettings()
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "traffic.collect.manual.ok", eventType: "traffic.collect.manual.ok",
sourceModule: "traffic", sourceModule: "traffic",
@@ -285,7 +284,7 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
}) })
} catch (error) { } catch (error) {
const msg = error instanceof Error ? error.message : String(error) const msg = error instanceof Error ? error.message : String(error)
appendEvent({ await appendEvent({
level: "critical", level: "critical",
eventType: "traffic.collect.manual.failed", eventType: "traffic.collect.manual.failed",
sourceModule: "traffic", sourceModule: "traffic",
@@ -302,11 +301,11 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
const rangeEndMs = Date.now() const rangeEndMs = Date.now()
const rangeStartMs = rangeEndMs - minutes * 60_000 const rangeStartMs = rangeEndMs - minutes * 60_000
const sinceIso = new Date(rangeStartMs).toISOString() const sinceIso = new Date(rangeStartMs).toISOString()
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const data = allServers.map((s): TrafficServerDto => { const data = await Promise.all(allServers.map(async (s): Promise<TrafficServerDto> => {
const rows = readServerSamplesInRange(s.id, sinceIso) const rows = await readServerSamplesInRange(s.id, sinceIso)
return buildServerTraffic(s, snapshotStatus(s.id), rows, rangeStartMs, rangeEndMs) return buildServerTraffic(s, await snapshotStatus(s.id), rows, rangeStartMs, rangeEndMs)
}) }))
return reply.send({ servers: data }) return reply.send({ servers: data })
}) })
@@ -314,7 +313,7 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/traffic/servers/:id/live", async (req, reply) => { app.get("/traffic/servers/:id/live", async (req, reply) => {
const p = req.params as { id?: string | number } const p = req.params as { id?: string | number }
const q = req.query as { iface?: string } const q = req.query as { iface?: string }
const server = getEnabledServerById(p.id ?? "") const server = await getEnabledServerById(p.id ?? "")
if (!server || !server.enabled) return reply.status(404).send({ error: "Server not found" }) if (!server || !server.enabled) return reply.status(404).send({ error: "Server not found" })
const onlyInterface = q.iface && q.iface !== "__all__" ? q.iface : undefined const onlyInterface = q.iface && q.iface !== "__all__" ? q.iface : undefined
@@ -375,12 +374,12 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
const serverId = Number.parseInt(String(p.id ?? ""), 10) const serverId = Number.parseInt(String(p.id ?? ""), 10)
if (!Number.isFinite(serverId)) return reply.status(400).send({ error: "id is required" }) if (!Number.isFinite(serverId)) return reply.status(400).send({ error: "id is required" })
const allRows = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all() const allRows = await db.select().from(servers).where(eq(servers.id, serverId)).limit(1)
const server = allRows[0] const server = allRows[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
const sinceIso = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString() const sinceIso = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString()
const rows = readServerSamplesInRange(serverId, sinceIso) const rows = await readServerSamplesInRange(serverId, sinceIso)
const byIface = new Map<string, typeof rows>() const byIface = new Map<string, typeof rows>()
for (const r of rows) { for (const r of rows) {
if (isLoopbackName(r.interfaceName)) continue if (isLoopbackName(r.interfaceName)) continue
@@ -413,17 +412,17 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
const q = req.query as { range?: string; iface?: string } const q = req.query as { range?: string; iface?: string }
const serverId = Number.parseInt(String(p.id ?? ""), 10) const serverId = Number.parseInt(String(p.id ?? ""), 10)
if (!Number.isFinite(serverId)) return reply.status(400).send({ error: "id is required" }) if (!Number.isFinite(serverId)) return reply.status(400).send({ error: "id is required" })
const server = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const server = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!server) return reply.status(404).send({ error: "Server not found" }) if (!server) return reply.status(404).send({ error: "Server not found" })
const minutes = rangeToMinutes(q.range) const minutes = rangeToMinutes(q.range)
const rangeEndMs = Date.now() const rangeEndMs = Date.now()
const rangeStartMs = rangeEndMs - minutes * 60_000 const rangeStartMs = rangeEndMs - minutes * 60_000
const sinceIso = new Date(rangeStartMs).toISOString() const sinceIso = new Date(rangeStartMs).toISOString()
const rows = readServerSamplesInRange(server.id, sinceIso) const rows = await readServerSamplesInRange(server.id, sinceIso)
const data = buildServerTraffic( const data = buildServerTraffic(
server, server,
snapshotStatus(server.id), await snapshotStatus(server.id),
rows, rows,
rangeStartMs, rangeStartMs,
rangeEndMs, rangeEndMs,
@@ -437,7 +436,7 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
const minutes = rangeToMinutes(q.range) const minutes = rangeToMinutes(q.range)
const rangeEndMs = Date.now() const rangeEndMs = Date.now()
const rangeStartMs = rangeEndMs - minutes * 60_000 const rangeStartMs = rangeEndMs - minutes * 60_000
return reply.send({ users: buildUserTrafficList(rangeStartMs, rangeEndMs) }) return reply.send({ users: await buildUserTrafficList(rangeStartMs, rangeEndMs) })
}) })
app.get("/traffic/users/:id", async (req, reply) => { app.get("/traffic/users/:id", async (req, reply) => {
@@ -448,7 +447,7 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
const minutes = rangeToMinutes(q.range) const minutes = rangeToMinutes(q.range)
const rangeEndMs = Date.now() const rangeEndMs = Date.now()
const rangeStartMs = rangeEndMs - minutes * 60_000 const rangeStartMs = rangeEndMs - minutes * 60_000
const users = buildUserTrafficList(rangeStartMs, rangeEndMs) const users = await buildUserTrafficList(rangeStartMs, rangeEndMs)
const user = users.find((u) => u.id === id) const user = users.find((u) => u.id === id)
if (!user) return reply.status(404).send({ error: "Пользователь не найден" }) if (!user) return reply.status(404).send({ error: "Пользователь не найден" })
return reply.send({ user }) return reply.send({ user })
@@ -459,7 +458,7 @@ const trafficRoutes: FastifyPluginAsyncZod = async (app) => {
const minutes = rangeToMinutes(q.range) const minutes = rangeToMinutes(q.range)
const rangeEndMs = Date.now() const rangeEndMs = Date.now()
const rangeStartMs = rangeEndMs - minutes * 60_000 const rangeStartMs = rangeEndMs - minutes * 60_000
return reply.send({ interfaces: buildBoundInterfaceTraffic(rangeStartMs, rangeEndMs) }) return reply.send({ interfaces: await buildBoundInterfaceTraffic(rangeStartMs, rangeEndMs) })
}) })
} }
+33 -33
View File
@@ -36,8 +36,8 @@ function toSeries(values: number[], target = 40): number[] {
return [...Array(target - values.length).fill(values[0] ?? 0), ...values] return [...Array(target - values.length).fill(values[0] ?? 0), ...values]
} }
/** Сэмпл ресурсов из SQLite (readResourceSamplesSince). */ /** Сэмпл ресурсов из PostgreSQL (readResourceSamplesSince). */
type ResourceSampleRow = ReturnType<typeof readResourceSamplesSince>[number] type ResourceSampleRow = Awaited<ReturnType<typeof readResourceSamplesSince>>[number]
/** /**
* Последний сэмпл по времени может быть offline после сбоя опроса тогда UI «пустой». * Последний сэмпл по времени может быть offline после сбоя опроса тогда UI «пустой».
@@ -67,7 +67,7 @@ function pickResourceDisplayRow(rows: ResourceSampleRow[], maxGapMs: number): {
const uptimeRoutes: FastifyPluginAsyncZod = async (app) => { const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/uptime/settings", async (_req, reply) => { app.get("/uptime/settings", async (_req, reply) => {
const s = readUptimeSettings() const s = await readUptimeSettings()
return reply.send({ return reply.send({
enabled: s.enabled, enabled: s.enabled,
resourcesEnabled: s.resourcesEnabled ?? s.enabled, resourcesEnabled: s.resourcesEnabled ?? s.enabled,
@@ -82,7 +82,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
lastCollectedAt: s.lastCollectedAt ?? null, lastCollectedAt: s.lastCollectedAt ?? null,
lastDurationMs: s.lastDurationMs ?? null, lastDurationMs: s.lastDurationMs ?? null,
lastError: s.lastError || null, lastError: s.lastError || null,
scheduler: getSchedulerStatus(), scheduler: await getSchedulerStatus(),
}) })
}) })
@@ -97,7 +97,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
speedIntervalSec?: number | string speedIntervalSec?: number | string
retentionDays?: number | string retentionDays?: number | string
} }
const updated = updateUptimeSettings({ const updated = await updateUptimeSettings({
enabled: body.enabled, enabled: body.enabled,
resourcesEnabled: body.resourcesEnabled, resourcesEnabled: body.resourcesEnabled,
pingEnabled: body.pingEnabled, pingEnabled: body.pingEnabled,
@@ -107,7 +107,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
speedIntervalSec: body.speedIntervalSec == null ? undefined : Math.max(10, Number.parseInt(String(body.speedIntervalSec), 10) || 60), speedIntervalSec: body.speedIntervalSec == null ? undefined : Math.max(10, Number.parseInt(String(body.speedIntervalSec), 10) || 60),
retentionDays: body.retentionDays == null ? undefined : Math.max(1, Number.parseInt(String(body.retentionDays), 10) || 14), retentionDays: body.retentionDays == null ? undefined : Math.max(1, Number.parseInt(String(body.retentionDays), 10) || 14),
}) })
refreshScheduler() await refreshScheduler()
return reply.send({ return reply.send({
ok: true, ok: true,
settings: { settings: {
@@ -122,7 +122,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
lastCollectedAt: updated.lastCollectedAt ?? null, lastCollectedAt: updated.lastCollectedAt ?? null,
lastDurationMs: updated.lastDurationMs ?? null, lastDurationMs: updated.lastDurationMs ?? null,
lastError: updated.lastError || null, lastError: updated.lastError || null,
scheduler: getSchedulerStatus(), scheduler: await getSchedulerStatus(),
}, },
}) })
}) })
@@ -130,7 +130,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
app.post("/uptime/collect-now", async (_req, reply) => { app.post("/uptime/collect-now", async (_req, reply) => {
await collectUptimeOnce() await collectUptimeOnce()
scheduleAlertEngineAfterDataCollectors() scheduleAlertEngineAfterDataCollectors()
const s = readUptimeSettings() const s = await readUptimeSettings()
return reply.send({ return reply.send({
ok: true, ok: true,
lastCollectedAt: s.lastCollectedAt ?? null, lastCollectedAt: s.lastCollectedAt ?? null,
@@ -158,7 +158,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
}) })
app.get("/uptime/probes", async (_req, reply) => { app.get("/uptime/probes", async (_req, reply) => {
const rows = readProbeRows() const rows = await readProbeRows()
return reply.send({ return reply.send({
probes: rows.map((p) => ({ probes: rows.map((p) => ({
id: p.id, id: p.id,
@@ -178,7 +178,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
}) })
app.get("/uptime/speed-probes", async (_req, reply) => { app.get("/uptime/speed-probes", async (_req, reply) => {
const rows = db.select().from(uptimeSpeedProbes).orderBy(asc(uptimeSpeedProbes.sortOrder)).all() const rows = await db.select().from(uptimeSpeedProbes).orderBy(asc(uptimeSpeedProbes.sortOrder))
return reply.send({ return reply.send({
probes: rows.map((r) => ({ probes: rows.map((r) => ({
id: r.id, id: r.id,
@@ -204,7 +204,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
}) })
app.get("/uptime/speed-test/runs", async (_req, reply) => { app.get("/uptime/speed-test/runs", async (_req, reply) => {
const rows = db.select().from(uptimeSpeedTestRuns).orderBy(desc(uptimeSpeedTestRuns.createdAt)).limit(200).all() const rows = await db.select().from(uptimeSpeedTestRuns).orderBy(desc(uptimeSpeedTestRuns.createdAt)).limit(200)
return reply.send({ return reply.send({
runs: rows.map((r) => ({ runs: rows.map((r) => ({
id: r.id, id: r.id,
@@ -263,12 +263,12 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
.filter((p) => Number.isFinite(p.srcServerId) && Number.isFinite(p.dstServerId) && p.srcServerId !== p.dstServerId) .filter((p) => Number.isFinite(p.srcServerId) && Number.isFinite(p.dstServerId) && p.srcServerId !== p.dstServerId)
const now = new Date().toISOString() const now = new Date().toISOString()
const existing = db.select({ id: uptimeSpeedProbes.id }).from(uptimeSpeedProbes).all() const existing = await db.select({ id: uptimeSpeedProbes.id }).from(uptimeSpeedProbes)
const nextIds = new Set(normalized.map((p) => p.id)) const nextIds = new Set(normalized.map((p) => p.id))
for (const row of existing) { for (const row of existing) {
if (nextIds.has(row.id)) continue if (nextIds.has(row.id)) continue
db.delete(uptimeSpeedProbes).where(eq(uptimeSpeedProbes.id, row.id)).run() await db.delete(uptimeSpeedProbes).where(eq(uptimeSpeedProbes.id, row.id))
} }
for (let i = 0; i < normalized.length; i += 1) { for (let i = 0; i < normalized.length; i += 1) {
@@ -285,15 +285,15 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
sortOrder: i, sortOrder: i,
updatedAt: now, updatedAt: now,
} }
const updated = db.update(uptimeSpeedProbes).set(patch).where(eq(uptimeSpeedProbes.id, p.id)).run() const updated = await db.update(uptimeSpeedProbes).set(patch).where(eq(uptimeSpeedProbes.id, p.id)).returning({ id: uptimeSpeedProbes.id })
if ((updated.changes ?? 0) > 0) continue if (updated.length > 0) continue
db.insert(uptimeSpeedProbes).values({ await db.insert(uptimeSpeedProbes).values({
id: p.id, id: p.id,
...patch, ...patch,
createdAt: now, createdAt: now,
}).run() })
} }
refreshScheduler() await refreshScheduler()
return reply.send({ ok: true }) return reply.send({ ok: true })
}) })
@@ -326,8 +326,8 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
: Number.parseInt(String(p.intervalSec), 10), : Number.parseInt(String(p.intervalSec), 10),
})) }))
.filter((p) => Number.isFinite(p.srcServerId) && p.name && p.target) .filter((p) => Number.isFinite(p.srcServerId) && p.name && p.target)
replaceProbes(normalized) await replaceProbes(normalized)
refreshScheduler() await refreshScheduler()
return reply.send({ ok: true }) return reply.send({ ok: true })
}) })
@@ -339,7 +339,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
if (typeof body.showOnDashboard !== "boolean") { if (typeof body.showOnDashboard !== "boolean") {
return reply.status(400).send({ error: "showOnDashboard boolean required" }) return reply.status(400).send({ error: "showOnDashboard boolean required" })
} }
const ok = updateProbeShowOnDashboard(probeId, body.showOnDashboard) const ok = await updateProbeShowOnDashboard(probeId, body.showOnDashboard)
if (!ok) return reply.status(404).send({ error: "Probe not found" }) if (!ok) return reply.status(404).send({ error: "Probe not found" })
return reply.send({ ok: true }) return reply.send({ ok: true })
}) })
@@ -348,7 +348,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
const params = req.params as { id?: string | number } const params = req.params as { id?: string | number }
const serverId = Number.parseInt(String(params.id ?? ""), 10) const serverId = Number.parseInt(String(params.id ?? ""), 10)
if (!Number.isFinite(serverId)) return reply.status(400).send({ error: "Invalid server id" }) if (!Number.isFinite(serverId)) return reply.status(400).send({ error: "Invalid server id" })
const srv = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const srv = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!srv) return reply.status(404).send({ error: "Server not found" }) if (!srv) return reply.status(404).send({ error: "Server not found" })
try { try {
const rows = await MikrotikClient.fromServer(srv).getInterfaces() const rows = await MikrotikClient.fromServer(srv).getInterfaces()
@@ -386,8 +386,8 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Invalid src/dst server id" }) return reply.status(400).send({ error: "Invalid src/dst server id" })
} }
if (srcId === dstId) return reply.status(400).send({ error: "Source and destination must be different" }) if (srcId === dstId) return reply.status(400).send({ error: "Source and destination must be different" })
const src = db.select().from(servers).where(eq(servers.id, srcId)).limit(1).all()[0] const src = (await db.select().from(servers).where(eq(servers.id, srcId)).limit(1))[0]
const dst = db.select().from(servers).where(eq(servers.id, dstId)).limit(1).all()[0] const dst = (await db.select().from(servers).where(eq(servers.id, dstId)).limit(1))[0]
if (!src || !dst) return reply.status(404).send({ error: "Server not found" }) if (!src || !dst) return reply.status(404).send({ error: "Server not found" })
try { try {
const result = await runBandwidthSpeedTest({ const result = await runBandwidthSpeedTest({
@@ -404,7 +404,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.send({ ok: true, result }) return reply.send({ ok: true, result })
} catch (e) { } catch (e) {
const raw = e instanceof Error ? e.message : "Bandwidth test failed" const raw = e instanceof Error ? e.message : "Bandwidth test failed"
recordSpeedTestFailure({ await recordSpeedTestFailure({
probeId: String(body.probeId ?? "").trim() || undefined, probeId: String(body.probeId ?? "").trim() || undefined,
runId: String(body.runId ?? "").trim() || undefined, runId: String(body.runId ?? "").trim() || undefined,
srcServerId: srcId, srcServerId: srcId,
@@ -430,7 +430,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
const params = req.params as { id?: string | number } const params = req.params as { id?: string | number }
const serverId = Number.parseInt(String(params.id ?? ""), 10) const serverId = Number.parseInt(String(params.id ?? ""), 10)
if (!Number.isFinite(serverId)) return reply.status(400).send({ error: "Invalid server id" }) if (!Number.isFinite(serverId)) return reply.status(400).send({ error: "Invalid server id" })
const srv = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const srv = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!srv) return reply.status(404).send({ error: "Server not found" }) if (!srv) return reply.status(404).send({ error: "Server not found" })
try { try {
const client = MikrotikClient.fromServer(srv) const client = MikrotikClient.fromServer(srv)
@@ -473,10 +473,10 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
const minutes = rangeToMinutes(q.range) const minutes = rangeToMinutes(q.range)
const sinceIso = new Date(Date.now() - minutes * 60_000).toISOString() const sinceIso = new Date(Date.now() - minutes * 60_000).toISOString()
const allServers = db.select().from(servers).all() const allServers = await db.select().from(servers)
const probeRows = readProbeRows() const probeRows = await readProbeRows()
const probeSamples = readProbeSamplesSince(sinceIso) const probeSamples = await readProbeSamplesSince(sinceIso)
const uptimeCfg = readUptimeSettings() const uptimeCfg = await readUptimeSettings()
const intervalSec = Math.max(15, uptimeCfg.intervalSec ?? 300) const intervalSec = Math.max(15, uptimeCfg.intervalSec ?? 300)
/** Допустимый разрыв между последней попыткой и последним «хорошим» сэмплом (не показывать данные часовой давности). */ /** Допустимый разрыв между последней попыткой и последним «хорошим» сэмплом (не показывать данные часовой давности). */
const resourceFallbackMaxGapMs = Math.min(2 * 3600 * 1000, Math.max(10 * 60 * 1000, intervalSec * 6 * 1000)) const resourceFallbackMaxGapMs = Math.min(2 * 3600 * 1000, Math.max(10 * 60 * 1000, intervalSec * 6 * 1000))
@@ -501,8 +501,8 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
} }
}) })
const resources = allServers.map((s) => { const resources = await Promise.all(allServers.map(async (s) => {
const rows = readResourceSamplesSince(sinceIso, s.id) const rows = await readResourceSamplesSince(sinceIso, s.id)
const { row: pick, hasData } = pickResourceDisplayRow(rows, resourceFallbackMaxGapMs) const { row: pick, hasData } = pickResourceDisplayRow(rows, resourceFallbackMaxGapMs)
const cpuHistory = toSeries(rows.map((r) => r.cpuLoad), 40) const cpuHistory = toSeries(rows.map((r) => r.cpuLoad), 40)
return { return {
@@ -518,7 +518,7 @@ const uptimeRoutes: FastifyPluginAsyncZod = async (app) => {
boardName: pick?.boardName || "RouterBOARD", boardName: pick?.boardName || "RouterBOARD",
temp: undefined as number | undefined, temp: undefined as number | undefined,
} }
}) }))
return reply.send({ probes, resources }) return reply.send({ probes, resources })
}) })
+7 -7
View File
@@ -31,7 +31,7 @@ function sendServiceError(reply: { status: (c: number) => { send: (b: unknown) =
const usersRoutes: FastifyPluginAsyncZod = async (app) => { const usersRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/users", async (_req, reply) => { app.get("/users", async (_req, reply) => {
return reply.send({ users: listUsers() }) return reply.send({ users: await listUsers() })
}) })
app.get("/users/interface-catalog", { app.get("/users/interface-catalog", {
@@ -49,7 +49,7 @@ const usersRoutes: FastifyPluginAsyncZod = async (app) => {
schema: { params: appUserIdParamSchema }, schema: { params: appUserIdParamSchema },
}, async (req, reply) => { }, async (req, reply) => {
const { id } = req.params as { id: string } const { id } = req.params as { id: string }
const user = getUserById(id) const user = await getUserById(id)
if (!user) return reply.status(404).send({ error: "Пользователь не найден" }) if (!user) return reply.status(404).send({ error: "Пользователь не найден" })
return reply.send({ user }) return reply.send({ user })
}) })
@@ -58,7 +58,7 @@ const usersRoutes: FastifyPluginAsyncZod = async (app) => {
schema: { body: appUserCreateSchema }, schema: { body: appUserCreateSchema },
}, async (req, reply) => { }, async (req, reply) => {
try { try {
const user = createUser(req.body as AppUserCreate) const user = await createUser(req.body as AppUserCreate)
return reply.status(201).send({ user }) return reply.status(201).send({ user })
} catch (err) { } catch (err) {
return sendServiceError(reply, err) return sendServiceError(reply, err)
@@ -70,7 +70,7 @@ const usersRoutes: FastifyPluginAsyncZod = async (app) => {
}, async (req, reply) => { }, async (req, reply) => {
const { id } = req.params as { id: string } const { id } = req.params as { id: string }
try { try {
const user = updateUser(id, req.body as AppUserUpdate) const user = await updateUser(id, req.body as AppUserUpdate)
return reply.send({ user }) return reply.send({ user })
} catch (err) { } catch (err) {
return sendServiceError(reply, err) return sendServiceError(reply, err)
@@ -82,7 +82,7 @@ const usersRoutes: FastifyPluginAsyncZod = async (app) => {
}, async (req, reply) => { }, async (req, reply) => {
const { id } = req.params as { id: string } const { id } = req.params as { id: string }
try { try {
deleteUser(id) await deleteUser(id)
return reply.status(204).send() return reply.status(204).send()
} catch (err) { } catch (err) {
return sendServiceError(reply, err) return sendServiceError(reply, err)
@@ -94,7 +94,7 @@ const usersRoutes: FastifyPluginAsyncZod = async (app) => {
}, async (req, reply) => { }, async (req, reply) => {
const { id } = req.params as { id: string } const { id } = req.params as { id: string }
try { try {
const binding = addBinding(id, req.body as UserBindingCreate) const binding = await addBinding(id, req.body as UserBindingCreate)
return reply.status(201).send({ binding }) return reply.status(201).send({ binding })
} catch (err) { } catch (err) {
return sendServiceError(reply, err) return sendServiceError(reply, err)
@@ -106,7 +106,7 @@ const usersRoutes: FastifyPluginAsyncZod = async (app) => {
}, async (req, reply) => { }, async (req, reply) => {
const { id, bindingId } = req.params as { id: string; bindingId: string } const { id, bindingId } = req.params as { id: string; bindingId: string }
try { try {
removeBinding(id, bindingId) await removeBinding(id, bindingId)
return reply.status(204).send() return reply.status(204).send()
} catch (err) { } catch (err) {
return sendServiceError(reply, err) return sendServiceError(reply, err)
+9 -9
View File
@@ -154,7 +154,7 @@ const wireguardRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = getEnabledServerById(body.serverId) const server = await getEnabledServerById(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
@@ -194,7 +194,7 @@ const wireguardRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const server = getEnabledServerById(serverIdParam(serverId)) const server = await getEnabledServerById(serverIdParam(serverId))
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
const d = parsed.data const d = parsed.data
@@ -219,7 +219,7 @@ const wireguardRoutes: FastifyPluginAsyncZod = async (app) => {
app.delete("/wireguard/interfaces/:serverId/:rosId", async (req, reply) => { app.delete("/wireguard/interfaces/:serverId/:rosId", async (req, reply) => {
const { serverId, rosId } = req.params as { serverId: string; rosId: string } const { serverId, rosId } = req.params as { serverId: string; rosId: string }
const server = getEnabledServerById(serverIdParam(serverId)) const server = await getEnabledServerById(serverIdParam(serverId))
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
try { try {
@@ -237,7 +237,7 @@ const wireguardRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = getEnabledServerById(body.serverId) const server = await getEnabledServerById(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
try { try {
@@ -255,7 +255,7 @@ const wireguardRoutes: FastifyPluginAsyncZod = async (app) => {
if (!parsed.success) { if (!parsed.success) {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const server = getEnabledServerById(serverIdParam(serverId)) const server = await getEnabledServerById(serverIdParam(serverId))
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const d = parsed.data const d = parsed.data
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
@@ -286,7 +286,7 @@ const wireguardRoutes: FastifyPluginAsyncZod = async (app) => {
app.delete("/wireguard/peers/:serverId/:rosId", async (req, reply) => { app.delete("/wireguard/peers/:serverId/:rosId", async (req, reply) => {
const { serverId, rosId } = req.params as { serverId: string; rosId: string } const { serverId, rosId } = req.params as { serverId: string; rosId: string }
const server = getEnabledServerById(serverIdParam(serverId)) const server = await getEnabledServerById(serverIdParam(serverId))
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
try { try {
@@ -315,7 +315,7 @@ const wireguardRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.send({ dryRun: true, preview }) return reply.send({ dryRun: true, preview })
} }
const server = getEnabledServerById(body.serverId) const server = await getEnabledServerById(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const client = MikrotikClient.fromServer(server) const client = MikrotikClient.fromServer(server)
try { try {
@@ -333,14 +333,14 @@ const wireguardRoutes: FastifyPluginAsyncZod = async (app) => {
return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() }) return reply.status(400).send({ error: "Некорректное тело запроса", details: parsed.error.flatten() })
} }
const body = parsed.data const body = parsed.data
const server = getEnabledServerById(body.serverId) const server = await getEnabledServerById(body.serverId)
if (!server) return reply.status(404).send({ error: "Сервер не найден" }) if (!server) return reply.status(404).send({ error: "Сервер не найден" })
const list = await listWireGuardInterfaces({ const list = await listWireGuardInterfaces({
serverId: String(server.id), serverId: String(server.id),
includePrivateKey: body.includePrivateKey === true, includePrivateKey: body.includePrivateKey === true,
}) })
const iface = findIface(list.interfaces, String(server.id), body.interfaceName) const iface = await findIface(list.interfaces, String(server.id), body.interfaceName)
if (!iface) return reply.status(404).send({ error: "Интерфейс не найден" }) if (!iface) return reply.status(404).send({ error: "Интерфейс не найден" })
if (body.format === "rsc") { if (body.format === "rsc") {
@@ -0,0 +1,20 @@
import { env } from "../config.js"
import { closePool, pool } from "../db/index.js"
import { applySqlMigrations } from "../db/migrate.js"
import { ensurePartitionsAround } from "../db/partitions.js"
import { importSqliteToPostgres } from "../db/sqlite-import.js"
const sqlitePath = process.argv.find((a) => a.startsWith("--sqlite="))?.slice(9) ?? env.DATABASE_PATH
const dryRun = process.argv.includes("--dry-run")
const fullHistory = process.argv.includes("--full-history")
const strict = process.argv.includes("--strict") || !dryRun
await applySqlMigrations(pool)
await ensurePartitionsAround(pool)
const report = await importSqliteToPostgres(pool, sqlitePath, { dryRun, strict, fullHistory })
console.log(JSON.stringify(report, null, 2))
if (!dryRun && report.rejects.length > 0 && strict) {
await closePool()
process.exit(1)
}
await closePool()
+9 -13
View File
@@ -85,15 +85,11 @@ async function deleteTxtRecord(token: string, zoneId: string, recordId: string):
await cloudflareRequest(token, `/zones/${zoneId}/dns_records/${recordId}`, { method: "DELETE" }) await cloudflareRequest(token, `/zones/${zoneId}/dns_records/${recordId}`, { method: "DELETE" })
} }
function parseWanUplinks(raw: string | null | undefined): WanUplinkRow[] { function parseWanUplinks(raw: unknown): WanUplinkRow[] {
if (!raw?.trim()) return [] const data = Array.isArray(raw) ? raw : typeof raw === "string" && raw.trim()
try { ? (() => { try { const p = JSON.parse(raw) as unknown; return Array.isArray(p) ? p : [] } catch { return [] } })()
const data = JSON.parse(raw) as unknown : []
if (!Array.isArray(data)) return [] return data.filter((row): row is WanUplinkRow => row != null && typeof row === "object")
return data.filter((row): row is WanUplinkRow => row != null && typeof row === "object")
} catch {
return []
}
} }
function normalizeIpv4(raw: string | undefined): string | null { function normalizeIpv4(raw: string | undefined): string | null {
@@ -217,10 +213,10 @@ async function sleep(ms: number) {
} }
async function getOrCreateAccountKey(): Promise<Buffer> { async function getOrCreateAccountKey(): Promise<Buffer> {
const existing = getAcmeAccountPrivateKey() const existing = await getAcmeAccountPrivateKey()
if (existing) return Buffer.from(existing) if (existing) return Buffer.from(existing)
const key = await acme.crypto.createPrivateKey() const key = await acme.crypto.createPrivateKey()
saveAcmeAccountPrivateKey(key.toString("utf8")) await saveAcmeAccountPrivateKey(key.toString("utf8"))
return key return key
} }
@@ -247,8 +243,8 @@ export async function issueCertificateWithCloudflareDns(params: {
trustStore: string[] trustStore: string[]
onStep?: (step: string) => void onStep?: (step: string) => void
}): Promise<void> { }): Promise<void> {
const settings = getAcmeSettingsPublic() const settings = await getAcmeSettingsPublic()
const token = getAcmeCloudflareToken() const token = await getAcmeCloudflareToken()
if (!token) throw new Error("Не настроен Cloudflare API token") if (!token) throw new Error("Не настроен Cloudflare API token")
const domains = [...new Set(params.domainNames.map((d) => d.trim().toLowerCase()).filter(Boolean))] const domains = [...new Set(params.domainNames.map((d) => d.trim().toLowerCase()).filter(Boolean))]
@@ -1,5 +1,5 @@
/** /**
* Внутренние хуки: после записи сигналов в SQLite (коллекторы) отложенный прогон `alert_engine`. * Внутренние хуки: после записи сигналов в PostgreSQL (коллекторы) отложенный прогон `alert_engine`.
* Не импортирует `scheduler.ts` (избегаем цикла); runner задаётся через `wireAlertEngineRunner`. * Не импортирует `scheduler.ts` (избегаем цикла); runner задаётся через `wireAlertEngineRunner`.
* *
* Любой код, пишущий таблицы для `buildSignalSnapshot`, после успешного коммита должен вызывать * Любой код, пишущий таблицы для `buildSignalSnapshot`, после успешного коммита должен вызывать
@@ -15,7 +15,7 @@ let alertAfterCollectChain: Promise<void> = Promise.resolve()
let alertEngineRunner: (() => Promise<void>) | null = null let alertEngineRunner: (() => Promise<void>) | null = null
/** Задать runner один раз при старте приложения (обычно `() => executeSchedulerJob("alert_engine")`). */ /** Задать runner один раз при старте приложения (обычно `() => await executeSchedulerJob("alert_engine")`). */
export function wireAlertEngineRunner(run: () => Promise<void>): void { export function wireAlertEngineRunner(run: () => Promise<void>): void {
alertEngineRunner = run alertEngineRunner = run
} }
@@ -39,7 +39,7 @@ async function runAlertEngineWhenIdle(): Promise<void> {
/** После успешной записи сигналов в БД коллекторами (не из `alert_engine`). */ /** После успешной записи сигналов в БД коллекторами (не из `alert_engine`). */
export function scheduleAlertEngineAfterDataCollectors(): void { export function scheduleAlertEngineAfterDataCollectors(): void {
if (alertAfterCollectTimer != null) clearTimeout(alertAfterCollectTimer) if (alertAfterCollectTimer != null) clearTimeout(alertAfterCollectTimer)
alertAfterCollectTimer = setTimeout(() => { alertAfterCollectTimer = setTimeout(async () => {
alertAfterCollectTimer = null alertAfterCollectTimer = null
alertAfterCollectChain = alertAfterCollectChain alertAfterCollectChain = alertAfterCollectChain
.then(() => runAlertEngineWhenIdle()) .then(() => runAlertEngineWhenIdle())
@@ -4,11 +4,10 @@ import { db } from "../../db/index.js"
import { alertEngineConfirmPending } from "../../db/schema.js" import { alertEngineConfirmPending } from "../../db/schema.js"
import type { ApiAlertRule } from "../alerts-service.js" import type { ApiAlertRule } from "../alerts-service.js"
import { computeStabilityReadyMap, deleteConfirmPending } from "./confirm-stability.js" import { computeStabilityReadyMap, deleteConfirmPending } from "./confirm-stability.js"
import { withPgOrSkip } from "../../test/pg.js"
const ruleId = "test-confirm-stability-recovery-bypass" const ruleId = "test-confirm-stability-recovery-bypass"
deleteConfirmPending(ruleId)
const baseRule: ApiAlertRule = { const baseRule: ApiAlertRule = {
id: ruleId, id: ruleId,
name: "Server transition", name: "Server transition",
@@ -46,25 +45,31 @@ const recoveryHit = {
payloadHash: "recovery-1", payloadHash: "recovery-1",
} }
if (!(await withPgOrSkip())) {
console.log("alert-engine confirm-stability tests skip")
process.exit(0)
}
await deleteConfirmPending(ruleId)
{ {
const m = computeStabilityReadyMap([baseRule], new Map([[ruleId, problemHit]])) const m = await computeStabilityReadyMap([baseRule], new Map([[ruleId, problemHit]]))
assert.equal(m.get(ruleId), false, "проблемный переход должен ждать confirmStabilitySec") assert.equal(m.get(ruleId), false, "проблемный переход должен ждать confirmStabilitySec")
} }
{ {
const m = computeStabilityReadyMap([baseRule], new Map([[ruleId, recoveryHit]])) const m = await computeStabilityReadyMap([baseRule], new Map([[ruleId, recoveryHit]]))
assert.equal(m.get(ruleId), true, "восстановление должно отправляться без задержки") assert.equal(m.get(ruleId), true, "восстановление должно отправляться без задержки")
} }
{ {
const m = computeStabilityReadyMap([baseRule], new Map([[ruleId, null]])) const m = await computeStabilityReadyMap([baseRule], new Map([[ruleId, null]]))
assert.equal(m.get(ruleId), false, "без hit отправка не готова") assert.equal(m.get(ruleId), false, "без hit отправка не готова")
const left = db const left = (await db
.select() .select()
.from(alertEngineConfirmPending) .from(alertEngineConfirmPending)
.where(eq(alertEngineConfirmPending.ruleId, ruleId)) .where(eq(alertEngineConfirmPending.ruleId, ruleId))
.limit(1) .limit(1))[0]
.all()[0]
assert.equal(left, undefined, "pending должен очищаться после отсутствия hit") assert.equal(left, undefined, "pending должен очищаться после отсутствия hit")
} }
@@ -4,46 +4,44 @@ import { alertEngineConfirmPending } from "../../db/schema.js"
import type { ApiAlertRule } from "../alerts-service.js" import type { ApiAlertRule } from "../alerts-service.js"
import type { RuleEvalHit } from "./types.js" import type { RuleEvalHit } from "./types.js"
export function deleteConfirmPending(ruleId: string) { export async function deleteConfirmPending(ruleId: string) {
db.delete(alertEngineConfirmPending).where(eq(alertEngineConfirmPending.ruleId, ruleId)).run() await db.delete(alertEngineConfirmPending).where(eq(alertEngineConfirmPending.ruleId, ruleId))
} }
/** /**
* Обновляет состояние ожидания стабильности и возвращает, можно ли отправлять уведомление в этом тике. * Обновляет состояние ожидания стабильности и возвращает, можно ли отправлять уведомление в этом тике.
* При отсутствии срабатывания (`hit == null`) ожидание сбрасывается. * При отсутствии срабатывания (`hit == null`) ожидание сбрасывается.
*/ */
export function stabilityReadyForSend( export async function stabilityReadyForSend(
ruleId: string, ruleId: string,
hit: RuleEvalHit | null, hit: RuleEvalHit | null,
delaySec: number | null | undefined, delaySec: number | null | undefined,
): boolean { ): Promise<boolean> {
if (!hit) { if (!hit) {
deleteConfirmPending(ruleId) await deleteConfirmPending(ruleId)
return false return false
} }
const sec = delaySec != null && delaySec > 0 ? Math.floor(delaySec) : 0 const sec = delaySec != null && delaySec > 0 ? Math.floor(delaySec) : 0
if (sec <= 0) { if (sec <= 0) {
deleteConfirmPending(ruleId) await deleteConfirmPending(ruleId)
return true return true
} }
const now = Date.now() const now = Date.now()
const nowIso = new Date().toISOString() const nowIso = new Date().toISOString()
const existing = db const existing = (await db
.select() .select()
.from(alertEngineConfirmPending) .from(alertEngineConfirmPending)
.where(eq(alertEngineConfirmPending.ruleId, ruleId)) .where(eq(alertEngineConfirmPending.ruleId, ruleId))
.limit(1) .limit(1))[0]
.all()[0]
if (!existing || existing.payloadHash !== hit.payloadHash) { if (!existing || existing.payloadHash !== hit.payloadHash) {
if (existing) { if (existing) {
db.update(alertEngineConfirmPending) await db.update(alertEngineConfirmPending)
.set({ payloadHash: hit.payloadHash, sinceAt: nowIso }) .set({ payloadHash: hit.payloadHash, sinceAt: nowIso })
.where(eq(alertEngineConfirmPending.ruleId, ruleId)) .where(eq(alertEngineConfirmPending.ruleId, ruleId))
.run()
} else { } else {
db.insert(alertEngineConfirmPending).values({ ruleId, payloadHash: hit.payloadHash, sinceAt: nowIso }).run() await db.insert(alertEngineConfirmPending).values({ ruleId, payloadHash: hit.payloadHash, sinceAt: nowIso })
} }
return false return false
} }
@@ -54,18 +52,18 @@ export function stabilityReadyForSend(
} }
/** После успешной отправки — сбросить таймер стабильности для следующего цикла. */ /** После успешной отправки — сбросить таймер стабильности для следующего цикла. */
export function clearConfirmPendingAfterSend(ruleIds: string[]) { export async function clearConfirmPendingAfterSend(ruleIds: string[]) {
for (const id of ruleIds) deleteConfirmPending(id) for (const id of ruleIds) await deleteConfirmPending(id)
} }
export function computeStabilityReadyMap( export async function computeStabilityReadyMap(
rules: ApiAlertRule[], rules: ApiAlertRule[],
hitByRule: Map<string, RuleEvalHit | null>, hitByRule: Map<string, RuleEvalHit | null>,
): Map<string, boolean> { ): Promise<Map<string, boolean>> {
const m = new Map<string, boolean>() const m = new Map<string, boolean>()
for (const rule of rules) { for (const rule of rules) {
if (!rule.enabled) { if (!rule.enabled) {
deleteConfirmPending(rule.id) await deleteConfirmPending(rule.id)
m.set(rule.id, false) m.set(rule.id, false)
continue continue
} }
@@ -76,7 +74,7 @@ export function computeStabilityReadyMap(
? rule.recoveryStabilitySec ? rule.recoveryStabilitySec
: 0 : 0
: rule.confirmStabilitySec : rule.confirmStabilitySec
m.set(rule.id, stabilityReadyForSend(rule.id, hit, delaySec)) m.set(rule.id, await stabilityReadyForSend(rule.id, hit, delaySec))
} }
return m return m
} }
@@ -73,20 +73,20 @@ export function buildRuleDiagnostics(
return out return out
} }
export function computeDecisions(args: { export async function computeDecisions(args: {
rules: ApiAlertRule[] rules: ApiAlertRule[]
groups: ApiAlertGroup[] groups: ApiAlertGroup[]
hitByRule: Map<string, RuleEvalHit | null> hitByRule: Map<string, RuleEvalHit | null>
state: EngineStateMap state: EngineStateMap
canSend: boolean canSend: boolean
}): { }): Promise<{
stabilityReady: Map<string, boolean> stabilityReady: Map<string, boolean>
standalone: StandaloneDecision[] standalone: StandaloneDecision[]
grouped: GroupDecision[] grouped: GroupDecision[]
ruleDiag: AlertEngineRuleDiag[] ruleDiag: AlertEngineRuleDiag[]
} { }> {
const { rules, groups, hitByRule, state, canSend } = args const { rules, groups, hitByRule, state, canSend } = args
const stabilityReady = computeStabilityReadyMap(rules, hitByRule) const stabilityReady = await computeStabilityReadyMap(rules, hitByRule)
const ruleDiag = buildRuleDiagnostics(rules, hitByRule, stabilityReady, state, canSend) const ruleDiag = buildRuleDiagnostics(rules, hitByRule, stabilityReady, state, canSend)
const standalone: StandaloneDecision[] = [] const standalone: StandaloneDecision[] = []
const grouped: GroupDecision[] = [] const grouped: GroupDecision[] = []
+15 -19
View File
@@ -17,7 +17,9 @@ export type AlertOutboxPayload = {
} }
} }
function safeParsePayload(raw: string): AlertOutboxPayload | null { function safeParsePayload(raw: unknown): AlertOutboxPayload | null {
if (raw && typeof raw === "object") return raw as AlertOutboxPayload
if (typeof raw !== "string") return null
try { try {
return JSON.parse(raw) as AlertOutboxPayload return JSON.parse(raw) as AlertOutboxPayload
} catch { } catch {
@@ -25,21 +27,20 @@ function safeParsePayload(raw: string): AlertOutboxPayload | null {
} }
} }
export function enqueueTelegramOutbox(input: { export async function enqueueTelegramOutbox(input: {
id: string id: string
dedupeKey: string dedupeKey: string
payload: AlertOutboxPayload payload: AlertOutboxPayload
maxRetries?: number maxRetries?: number
}): boolean { }): Promise<boolean> {
const existing = db const existing = (await db
.select() .select()
.from(alertOutbox) .from(alertOutbox)
.where(eq(alertOutbox.dedupeKey, input.dedupeKey)) .where(eq(alertOutbox.dedupeKey, input.dedupeKey))
.limit(1) .limit(1))[0]
.all()[0]
if (existing && (existing.status === "pending" || existing.status === "sent")) return false if (existing && (existing.status === "pending" || existing.status === "sent")) return false
const nowIso = new Date().toISOString() const nowIso = new Date().toISOString()
db.insert(alertOutbox) await db.insert(alertOutbox)
.values({ .values({
id: input.id, id: input.id,
dedupeKey: input.dedupeKey, dedupeKey: input.dedupeKey,
@@ -48,22 +49,20 @@ export function enqueueTelegramOutbox(input: {
retryCount: 0, retryCount: 0,
maxRetries: Math.max(1, Math.floor(input.maxRetries ?? 3)), maxRetries: Math.max(1, Math.floor(input.maxRetries ?? 3)),
nextAttemptAt: nowIso, nextAttemptAt: nowIso,
payloadJson: JSON.stringify(input.payload), payloadJson: input.payload,
}) })
.run()
return true return true
} }
export function dispatchPendingOutbox(limit = 25): Promise<{ sent: number; failed: number; errors: string[] }> { export function dispatchPendingOutbox(limit = 25): Promise<{ sent: number; failed: number; errors: string[] }> {
return (async () => { return (async () => {
const nowIso = new Date().toISOString() const nowIso = new Date().toISOString()
const rows = db const rows = await db
.select() .select()
.from(alertOutbox) .from(alertOutbox)
.where(and(eq(alertOutbox.status, "pending"), lte(alertOutbox.nextAttemptAt, nowIso))) .where(and(eq(alertOutbox.status, "pending"), lte(alertOutbox.nextAttemptAt, nowIso)))
.orderBy(asc(alertOutbox.createdAt)) .orderBy(asc(alertOutbox.createdAt))
.limit(Math.max(1, limit)) .limit(Math.max(1, limit))
.all()
let sent = 0 let sent = 0
let failed = 0 let failed = 0
@@ -72,27 +71,25 @@ export function dispatchPendingOutbox(limit = 25): Promise<{ sent: number; faile
for (const row of rows) { for (const row of rows) {
const payload = safeParsePayload(row.payloadJson) const payload = safeParsePayload(row.payloadJson)
if (!payload) { if (!payload) {
db.update(alertOutbox) await db.update(alertOutbox)
.set({ .set({
status: "failed", status: "failed",
lastError: "invalid payload_json", lastError: "invalid payload_json",
}) })
.where(eq(alertOutbox.id, row.id)) .where(eq(alertOutbox.id, row.id))
.run()
failed += 1 failed += 1
continue continue
} }
const send = await sendTelegramAlertMessage({ text: payload.text, chatId: payload.chatId }) const send = await sendTelegramAlertMessage({ text: payload.text, chatId: payload.chatId })
if (send.ok) { if (send.ok) {
db.update(alertOutbox) await db.update(alertOutbox)
.set({ .set({
status: "sent", status: "sent",
sentAt: new Date().toISOString(), sentAt: new Date().toISOString(),
lastError: null, lastError: null,
}) })
.where(eq(alertOutbox.id, row.id)) .where(eq(alertOutbox.id, row.id))
.run() await appendAlertHistory({
appendAlertHistory({
id: payload.history.id, id: payload.history.id,
ruleId: payload.history.ruleId ?? null, ruleId: payload.history.ruleId ?? null,
groupId: payload.history.groupId ?? null, groupId: payload.history.groupId ?? null,
@@ -107,7 +104,7 @@ export function dispatchPendingOutbox(limit = 25): Promise<{ sent: number; faile
} }
const nextRetry = row.retryCount + 1 const nextRetry = row.retryCount + 1
const exhausted = nextRetry >= row.maxRetries const exhausted = nextRetry >= row.maxRetries
db.update(alertOutbox) await db.update(alertOutbox)
.set({ .set({
retryCount: nextRetry, retryCount: nextRetry,
status: exhausted ? "failed" : "pending", status: exhausted ? "failed" : "pending",
@@ -115,9 +112,8 @@ export function dispatchPendingOutbox(limit = 25): Promise<{ sent: number; faile
lastError: send.error, lastError: send.error,
}) })
.where(eq(alertOutbox.id, row.id)) .where(eq(alertOutbox.id, row.id))
.run()
if (exhausted) { if (exhausted) {
appendAlertHistory({ await appendAlertHistory({
id: payload.history.id, id: payload.history.id,
ruleId: payload.history.ruleId ?? null, ruleId: payload.history.ruleId ?? null,
groupId: payload.history.groupId ?? null, groupId: payload.history.groupId ?? null,
@@ -1,5 +1,6 @@
import { eq } from "drizzle-orm" import { eq } from "drizzle-orm"
import { db } from "../../db/index.js" import { db } from "../../db/index.js"
import { parseJsonObject } from "../../db/json.js"
import { alertEnginePrevLive } from "../../db/schema.js" import { alertEnginePrevLive } from "../../db/schema.js"
export type PrevLiveKind = "gre" | "bgp" export type PrevLiveKind = "gre" | "bgp"
@@ -7,42 +8,32 @@ export type PrevLiveKind = "gre" | "bgp"
/** Предыдущие строковые состояния (GRE: up|down|degraded; BGP: state как с роутера). */ /** Предыдущие строковые состояния (GRE: up|down|degraded; BGP: state как с роутера). */
export type PrevLiveStringMap = Record<string, string> export type PrevLiveStringMap = Record<string, string>
function safeParseMap(json: string | null | undefined): PrevLiveStringMap { function safeParseMap(json: unknown): PrevLiveStringMap {
if (!json || json.trim() === "") return {} return parseJsonObject(json) as PrevLiveStringMap
try {
const v = JSON.parse(json) as unknown
if (v == null || typeof v !== "object" || Array.isArray(v)) return {}
return v as PrevLiveStringMap
} catch {
return {}
}
} }
export function loadPrevLiveMap(kind: PrevLiveKind): PrevLiveStringMap { export async function loadPrevLiveMap(kind: PrevLiveKind): Promise<PrevLiveStringMap> {
const row = db const row = (await db
.select() .select()
.from(alertEnginePrevLive) .from(alertEnginePrevLive)
.where(eq(alertEnginePrevLive.kind, kind)) .where(eq(alertEnginePrevLive.kind, kind))
.limit(1) .limit(1))[0]
.all()[0]
return safeParseMap(row?.payloadJson) return safeParseMap(row?.payloadJson)
} }
export function savePrevLiveMap(kind: PrevLiveKind, map: PrevLiveStringMap) { export async function savePrevLiveMap(kind: PrevLiveKind, map: PrevLiveStringMap) {
const payloadJson = JSON.stringify(map) const payloadJson = map
const existing = db const existing = (await db
.select() .select()
.from(alertEnginePrevLive) .from(alertEnginePrevLive)
.where(eq(alertEnginePrevLive.kind, kind)) .where(eq(alertEnginePrevLive.kind, kind))
.limit(1) .limit(1))[0]
.all()[0]
if (existing) { if (existing) {
if (existing.payloadJson === payloadJson) return if (existing.payloadJson === payloadJson) return
db.update(alertEnginePrevLive) await db.update(alertEnginePrevLive)
.set({ payloadJson, updatedAt: new Date().toISOString() }) .set({ payloadJson, updatedAt: new Date().toISOString() })
.where(eq(alertEnginePrevLive.kind, kind)) .where(eq(alertEnginePrevLive.kind, kind))
.run()
return return
} }
db.insert(alertEnginePrevLive).values({ kind, payloadJson, updatedAt: new Date().toISOString() }).run() await db.insert(alertEnginePrevLive).values({ kind, payloadJson, updatedAt: new Date().toISOString() })
} }
+24 -25
View File
@@ -26,7 +26,7 @@ const SNAPSHOT_SOURCE_WAIT_MS = 30_000
const SNAPSHOT_SOURCE_POLL_MS = 20 const SNAPSHOT_SOURCE_POLL_MS = 20
/** /**
* Не строить снимок, пока коллекторы пишут в SQLite иначе гонка с `alert_engine` по таймеру * Не строить снимок, пока коллекторы пишут в PostgreSQL иначе гонка с `alert_engine` по таймеру
* (в т.ч. слот планировщика занят до первого `await` в коллекторе, когда внутренний `collecting` ещё false). * (в т.ч. слот планировщика занят до первого `await` в коллекторе, когда внутренний `collecting` ещё false).
*/ */
async function awaitSnapshotSourcesIdle(): Promise<void> { async function awaitSnapshotSourcesIdle(): Promise<void> {
@@ -52,8 +52,8 @@ function newOutboxId(): string {
return `ao-${Date.now()}-${Math.random().toString(36).slice(2, 10)}` return `ao-${Date.now()}-${Math.random().toString(36).slice(2, 10)}`
} }
function loadEngineStateMap(): Map<string, { lastFiredAt: string; lastPayloadHash: string | null }> { async function loadEngineStateMap(): Promise<Map<string, { lastFiredAt: string; lastPayloadHash: string | null }>> {
const rows = db.select().from(alertEngineState).all() const rows = await db.select().from(alertEngineState)
const m = new Map<string, { lastFiredAt: string; lastPayloadHash: string | null }>() const m = new Map<string, { lastFiredAt: string; lastPayloadHash: string | null }>()
for (const r of rows) { for (const r of rows) {
m.set(r.scopeKey, { lastFiredAt: r.lastFiredAt, lastPayloadHash: r.lastPayloadHash ?? null }) m.set(r.scopeKey, { lastFiredAt: r.lastFiredAt, lastPayloadHash: r.lastPayloadHash ?? null })
@@ -61,15 +61,14 @@ function loadEngineStateMap(): Map<string, { lastFiredAt: string; lastPayloadHas
return m return m
} }
function upsertEngineState(scopeKey: string, lastFiredAt: string, lastPayloadHash: string | null) { async function upsertEngineState(scopeKey: string, lastFiredAt: string, lastPayloadHash: string | null) {
const existing = db.select().from(alertEngineState).where(eq(alertEngineState.scopeKey, scopeKey)).limit(1).all()[0] const existing = (await db.select().from(alertEngineState).where(eq(alertEngineState.scopeKey, scopeKey)).limit(1))[0]
if (existing) { if (existing) {
db.update(alertEngineState) await db.update(alertEngineState)
.set({ lastFiredAt, lastPayloadHash }) .set({ lastFiredAt, lastPayloadHash })
.where(eq(alertEngineState.scopeKey, scopeKey)) .where(eq(alertEngineState.scopeKey, scopeKey))
.run()
} else { } else {
db.insert(alertEngineState).values({ scopeKey, lastFiredAt, lastPayloadHash }).run() await db.insert(alertEngineState).values({ scopeKey, lastFiredAt, lastPayloadHash })
} }
} }
@@ -90,16 +89,16 @@ function ruleScopeKey(ruleId: string, transition?: RuleEvalHit["transition"]): s
export async function runAlertEngineOnce(): Promise<AlertEngineRunResult> { export async function runAlertEngineOnce(): Promise<AlertEngineRunResult> {
const errors: string[] = [] const errors: string[] = []
const sampledAt = new Date().toISOString() const sampledAt = new Date().toISOString()
const latestSourceFinishedAt = getLatestSourceFinishedAt() const latestSourceFinishedAt = await getLatestSourceFinishedAt()
const prevSourceFinishedAt = getSourceWatermark() const prevSourceFinishedAt = await getSourceWatermark()
const hasNewSources = const hasNewSources =
latestSourceFinishedAt != null && latestSourceFinishedAt != null &&
(prevSourceFinishedAt == null || latestSourceFinishedAt > prevSourceFinishedAt) (prevSourceFinishedAt == null || latestSourceFinishedAt > prevSourceFinishedAt)
const rules = listAlertRules() const rules = await listAlertRules()
const groups = listAlertGroups() const groups = await listAlertGroups()
const state = loadEngineStateMap() const state = await loadEngineStateMap()
const pub = getTelegramPublic() const pub = await getTelegramPublic()
const canSend = pub.tokenConfigured && Boolean(pub.chatId?.trim()) const canSend = pub.tokenConfigured && Boolean(pub.chatId?.trim())
let standaloneFires = 0 let standaloneFires = 0
let groupFires = 0 let groupFires = 0
@@ -107,10 +106,10 @@ export async function runAlertEngineOnce(): Promise<AlertEngineRunResult> {
let ruleDiag: AlertEngineRunResult["ruleDiag"] = [] let ruleDiag: AlertEngineRunResult["ruleDiag"] = []
if (hasNewSources) { if (hasNewSources) {
await awaitSnapshotSourcesIdle() await awaitSnapshotSourcesIdle()
const snap = ingestAlertSignals() const snap = await ingestAlertSignals()
const matched = matchRules(rules, snap) const matched = matchRules(rules, snap)
errors.push(...matched.errors) errors.push(...matched.errors)
const { standalone, grouped, ruleDiag: diag } = computeDecisions({ const { standalone, grouped, ruleDiag: diag } = await computeDecisions({
rules, rules,
groups, groups,
hitByRule: matched.hitByRule, hitByRule: matched.hitByRule,
@@ -123,7 +122,7 @@ export async function runAlertEngineOnce(): Promise<AlertEngineRunResult> {
if (!canSend) break if (!canSend) break
const firedAt = new Date().toISOString() const firedAt = new Date().toISOString()
const key = ruleScopeKey(d.rule.id, d.hit.transition) const key = ruleScopeKey(d.rule.id, d.hit.transition)
const queued = enqueueTelegramOutbox({ const queued = await enqueueTelegramOutbox({
id: newOutboxId(), id: newOutboxId(),
dedupeKey: `${key}:${d.hit.payloadHash}`, dedupeKey: `${key}:${d.hit.payloadHash}`,
payload: { payload: {
@@ -142,8 +141,8 @@ export async function runAlertEngineOnce(): Promise<AlertEngineRunResult> {
}) })
if (!queued) continue if (!queued) continue
standaloneFires += 1 standaloneFires += 1
clearConfirmPendingAfterSend([d.rule.id]) await clearConfirmPendingAfterSend([d.rule.id])
upsertEngineState(key, firedAt, d.hit.payloadHash) await upsertEngineState(key, firedAt, d.hit.payloadHash)
state.set(key, { lastFiredAt: firedAt, lastPayloadHash: d.hit.payloadHash }) state.set(key, { lastFiredAt: firedAt, lastPayloadHash: d.hit.payloadHash })
} }
@@ -155,7 +154,7 @@ export async function runAlertEngineOnce(): Promise<AlertEngineRunResult> {
const body = `Группа «${d.group.name}» (${d.group.combineMode === "any" ? "ANY" : "ALL"})\n\n${lines.join("\n")}` const body = `Группа «${d.group.name}» (${d.group.combineMode === "any" ? "ANY" : "ALL"})\n\n${lines.join("\n")}`
const hash = d.hits.map((h) => h.payloadHash).sort().join("|") const hash = d.hits.map((h) => h.payloadHash).sort().join("|")
const key = `group:${d.group.id}` const key = `group:${d.group.id}`
const queued = enqueueTelegramOutbox({ const queued = await enqueueTelegramOutbox({
id: newOutboxId(), id: newOutboxId(),
dedupeKey: `${key}:${hash}`, dedupeKey: `${key}:${hash}`,
payload: { payload: {
@@ -173,18 +172,18 @@ export async function runAlertEngineOnce(): Promise<AlertEngineRunResult> {
}) })
if (!queued) continue if (!queued) continue
groupFires += 1 groupFires += 1
upsertEngineState(key, firedAt, hash) await upsertEngineState(key, firedAt, hash)
state.set(key, { lastFiredAt: firedAt, lastPayloadHash: hash }) state.set(key, { lastFiredAt: firedAt, lastPayloadHash: hash })
clearConfirmPendingAfterSend(d.members.map((r) => r.id)) await clearConfirmPendingAfterSend(d.members.map((r) => r.id))
} }
updateSourceWatermark(latestSourceFinishedAt) await updateSourceWatermark(latestSourceFinishedAt)
} }
const outbox = await dispatchPendingOutbox() const outbox = await dispatchPendingOutbox()
errors.push(...outbox.errors) errors.push(...outbox.errors)
if (standaloneFires > 0 || groupFires > 0) { if (standaloneFires > 0 || groupFires > 0) {
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "alerts.engine.fired", eventType: "alerts.engine.fired",
sourceModule: "alerts", sourceModule: "alerts",
@@ -197,7 +196,7 @@ export async function runAlertEngineOnce(): Promise<AlertEngineRunResult> {
}) })
} }
if (errors.length > 0) { if (errors.length > 0) {
appendEvent({ await appendEvent({
level: "warning", level: "warning",
eventType: "alerts.engine.errors", eventType: "alerts.engine.errors",
sourceModule: "alerts", sourceModule: "alerts",
@@ -1,6 +1,6 @@
import { buildSignalSnapshotFromCollectors } from "./signals-from-collectors.js" import { buildSignalSnapshotFromCollectors } from "./signals-from-collectors.js"
/** Единая точка входа для подготовки сигналов к rule matching. */ /** Единая точка входа для подготовки сигналов к rule matching. */
export function ingestAlertSignals() { export async function ingestAlertSignals() {
return buildSignalSnapshotFromCollectors() return await buildSignalSnapshotFromCollectors()
} }
@@ -1,4 +1,4 @@
import { sqliteDatabase } from "../../db/index.js" import { dbAll } from "../../db/index.js"
import type { import type {
GreBgpSnapshotRunSnapshot, GreBgpSnapshotRunSnapshot,
PingRunSnapshot, PingRunSnapshot,
@@ -20,25 +20,23 @@ function normalizeNameKey(v: string): string {
return v.trim().toLowerCase() return v.trim().toLowerCase()
} }
function safeParseRunSnapshot(raw: string | null): SchedulerRunSnapshot | null { function safeParseRunSnapshot(raw: unknown): SchedulerRunSnapshot | null {
if (!raw) return null const obj = typeof raw === "string" ? (() => {
try { try { return JSON.parse(raw) as unknown } catch { return null }
return JSON.parse(raw) as SchedulerRunSnapshot })() : raw
} catch { if (!obj || typeof obj !== "object") return null
return null return obj as SchedulerRunSnapshot
}
} }
function readLatestOkRunSnapshots(jobKey: string, limit = 32): SchedulerRunSnapshot[] { async function readLatestOkRunSnapshots(jobKey: string, limit = 32): Promise<SchedulerRunSnapshot[]> {
const rows = sqliteDatabase const rows = await dbAll<{ resultJson: unknown }>(
.prepare( `SELECT result_json AS "resultJson"
`SELECT result_json AS resultJson FROM scheduler_runs
FROM scheduler_runs WHERE job_key = ? AND status = 'ok' AND result_json IS NOT NULL
WHERE job_key = ? AND status = 'ok' AND result_json IS NOT NULL ORDER BY finished_at DESC
ORDER BY finished_at DESC LIMIT ?`,
LIMIT ?`, [jobKey, limit],
) )
.all(jobKey, limit) as { resultJson: string | null }[]
const out: SchedulerRunSnapshot[] = [] const out: SchedulerRunSnapshot[] = []
for (const row of rows) { for (const row of rows) {
const parsed = safeParseRunSnapshot(row.resultJson) const parsed = safeParseRunSnapshot(row.resultJson)
@@ -227,13 +225,13 @@ function collectBgpSignalsFromRuns(runs: GreBgpSnapshotRunSnapshot[]): BgpPeerSi
} }
/** Собирает сигналы alert_engine напрямую из результатов collector jobs (`scheduler_runs.result_json`). */ /** Собирает сигналы alert_engine напрямую из результатов collector jobs (`scheduler_runs.result_json`). */
export function buildSignalSnapshotFromCollectors(): SignalSnapshot { export async function buildSignalSnapshotFromCollectors(): Promise<SignalSnapshot> {
const sampledAt = new Date().toISOString() const sampledAt = new Date().toISOString()
const resourceRuns = readLatestOkRunSnapshots("uptime_resources", 16) as ResourcesRunSnapshot[] const resourceRuns = await readLatestOkRunSnapshots("uptime_resources", 16) as ResourcesRunSnapshot[]
const restRuns = readLatestOkRunSnapshots("servers_rest_ping", 16) as ServersRestPingRunSnapshot[] const restRuns = await readLatestOkRunSnapshots("servers_rest_ping", 16) as ServersRestPingRunSnapshot[]
const pingRuns = readLatestOkRunSnapshots("uptime_ping", 24) as PingRunSnapshot[] const pingRuns = await readLatestOkRunSnapshots("uptime_ping", 24) as PingRunSnapshot[]
const trafficRuns = readLatestOkRunSnapshots("traffic", 8) as TrafficRunSnapshot[] const trafficRuns = await readLatestOkRunSnapshots("traffic", 8) as TrafficRunSnapshot[]
const greBgpRuns = readLatestOkRunSnapshots("gre_bgp", 8) as GreBgpSnapshotRunSnapshot[] const greBgpRuns = await readLatestOkRunSnapshots("gre_bgp", 8) as GreBgpSnapshotRunSnapshot[]
const resourceByKey = collectServerSignalsFromResourceRuns(resourceRuns) const resourceByKey = collectServerSignalsFromResourceRuns(resourceRuns)
const restByKey = collectServerSignalsFromRestRuns(restRuns) const restByKey = collectServerSignalsFromRestRuns(restRuns)
@@ -2,6 +2,6 @@ import { buildSignalSnapshotFromCollectors } from "./signals-from-collectors.js"
import type { SignalSnapshot } from "./types.js" import type { SignalSnapshot } from "./types.js"
/** Legacy-совместимость: единый источник сигналов — collector snapshots из scheduler_runs. */ /** Legacy-совместимость: единый источник сигналов — collector snapshots из scheduler_runs. */
export function buildSignalSnapshot(): SignalSnapshot { export async function buildSignalSnapshot(): Promise<SignalSnapshot> {
return buildSignalSnapshotFromCollectors() return await buildSignalSnapshotFromCollectors()
} }
@@ -11,44 +11,40 @@ const SOURCE_JOBS = [
"gre_bgp", "gre_bgp",
] as const ] as const
export function getLatestSourceFinishedAt(): string | null { export async function getLatestSourceFinishedAt(): Promise<string | null> {
const row = db const row = (await db
.select({ finishedAt: schedulerRuns.finishedAt }) .select({ finishedAt: schedulerRuns.finishedAt })
.from(schedulerRuns) .from(schedulerRuns)
.where(inArray(schedulerRuns.jobKey, [...SOURCE_JOBS])) .where(inArray(schedulerRuns.jobKey, [...SOURCE_JOBS]))
.orderBy(desc(schedulerRuns.finishedAt)) .orderBy(desc(schedulerRuns.finishedAt))
.limit(1) .limit(1))[0]
.all()[0]
return row?.finishedAt ?? null return row?.finishedAt ?? null
} }
export function getSourceWatermark(): string | null { export async function getSourceWatermark(): Promise<string | null> {
const row = db.select().from(alertEngineCursor).where(eq(alertEngineCursor.id, 1)).limit(1).all()[0] const row = (await db.select().from(alertEngineCursor).where(eq(alertEngineCursor.id, 1)).limit(1))[0]
return row?.lastSourceFinishedAt ?? null return row?.lastSourceFinishedAt ?? null
} }
export function updateSourceWatermark(lastSourceFinishedAt: string | null): void { export async function updateSourceWatermark(lastSourceFinishedAt: string | null): Promise<void> {
const existing = db const existing = (await db
.select() .select()
.from(alertEngineCursor) .from(alertEngineCursor)
.where(eq(alertEngineCursor.id, 1)) .where(eq(alertEngineCursor.id, 1))
.limit(1) .limit(1))[0]
.all()[0]
if (existing) { if (existing) {
db.update(alertEngineCursor) await db.update(alertEngineCursor)
.set({ .set({
lastSourceFinishedAt, lastSourceFinishedAt,
updatedAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
}) })
.where(eq(alertEngineCursor.id, 1)) .where(eq(alertEngineCursor.id, 1))
.run()
return return
} }
db.insert(alertEngineCursor) await db.insert(alertEngineCursor)
.values({ .values({
id: 1, id: 1,
lastSourceFinishedAt, lastSourceFinishedAt,
updatedAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
}) })
.run()
} }
+107 -116
View File
@@ -1,5 +1,5 @@
import { asc, desc, eq, sql } from "drizzle-orm" import { asc, desc, eq, sql } from "drizzle-orm"
import { db, sqliteDatabase } from "../db/index.js" import { db, dbAll } from "../db/index.js"
import { import {
alertGroups, alertGroups,
alertHistory, alertHistory,
@@ -55,24 +55,22 @@ export type ApiAlertHistoryEntry = {
source: "db" | "live" source: "db" | "live"
} }
function ensureTelegramRow() { async function ensureTelegramRow() {
let row = db.select().from(alertTelegramSettings).where(eq(alertTelegramSettings.id, 1)).limit(1).all()[0] let row = (await db.select().from(alertTelegramSettings).where(eq(alertTelegramSettings.id, 1)).limit(1))[0]
if (!row) { if (!row) {
db.insert(alertTelegramSettings).values({ id: 1 }).run() await db.insert(alertTelegramSettings).values({ id: 1 })
row = db.select().from(alertTelegramSettings).where(eq(alertTelegramSettings.id, 1)).limit(1).all()[0] row = (await db.select().from(alertTelegramSettings).where(eq(alertTelegramSettings.id, 1)).limit(1))[0]
} }
return row! return row!
} }
function lastFiredByRuleId(): Map<string, string> { async function lastFiredByRuleId(): Promise<Map<string, string>> {
const rows = sqliteDatabase const rows = await dbAll<{ ruleId: string; mx: string }>(
.prepare( `SELECT rule_id AS "ruleId", MAX(fired_at) AS mx
`SELECT rule_id AS ruleId, MAX(fired_at) AS mx FROM alert_history
FROM alert_history WHERE rule_id IS NOT NULL AND rule_id != ''
WHERE rule_id IS NOT NULL AND rule_id != '' GROUP BY rule_id`,
GROUP BY rule_id`, )
)
.all() as { ruleId: string; mx: string }[]
const m = new Map<string, string>() const m = new Map<string, string>()
for (const r of rows) { for (const r of rows) {
if (r.ruleId && r.mx) m.set(r.ruleId, r.mx) if (r.ruleId && r.mx) m.set(r.ruleId, r.mx)
@@ -80,12 +78,11 @@ function lastFiredByRuleId(): Map<string, string> {
return m return m
} }
function targetsByRuleId(): Map<string, string[]> { async function targetsByRuleId(): Promise<Map<string, string[]>> {
const rows = db const rows = await db
.select() .select()
.from(alertRuleTargets) .from(alertRuleTargets)
.orderBy(asc(alertRuleTargets.sortIndex), asc(alertRuleTargets.id)) .orderBy(asc(alertRuleTargets.sortIndex), asc(alertRuleTargets.id))
.all()
const m = new Map<string, string[]>() const m = new Map<string, string[]>()
for (const t of rows) { for (const t of rows) {
const rid = t.ruleId const rid = t.ruleId
@@ -96,12 +93,11 @@ function targetsByRuleId(): Map<string, string[]> {
return m return m
} }
function conditionsByRuleId(): Map<string, string[]> { async function conditionsByRuleId(): Promise<Map<string, string[]>> {
const rows = db const rows = await db
.select() .select()
.from(alertRuleConditions) .from(alertRuleConditions)
.orderBy(asc(alertRuleConditions.sortIndex), asc(alertRuleConditions.id)) .orderBy(asc(alertRuleConditions.sortIndex), asc(alertRuleConditions.id))
.all()
const m = new Map<string, string[]>() const m = new Map<string, string[]>()
for (const t of rows) { for (const t of rows) {
const rid = t.ruleId const rid = t.ruleId
@@ -112,26 +108,25 @@ function conditionsByRuleId(): Map<string, string[]> {
return m return m
} }
export function listAlertGroups(): ApiAlertGroup[] { export async function listAlertGroups(): Promise<ApiAlertGroup[]> {
return db const rows = await db
.select() .select()
.from(alertGroups) .from(alertGroups)
.orderBy(asc(alertGroups.name)) .orderBy(asc(alertGroups.name))
.all() return rows.map((g) => ({
.map((g) => ({ id: g.id,
id: g.id, name: g.name,
name: g.name, combineMode: g.combineMode,
combineMode: g.combineMode, enabled: g.enabled,
enabled: g.enabled, cooldownOverride: g.cooldownOverride ?? null,
cooldownOverride: g.cooldownOverride ?? null, }))
}))
} }
export function listAlertRules(): ApiAlertRule[] { export async function listAlertRules(): Promise<ApiAlertRule[]> {
const lf = lastFiredByRuleId() const lf = await lastFiredByRuleId()
const tgtMap = targetsByRuleId() const tgtMap = await targetsByRuleId()
const condMap = conditionsByRuleId() const condMap = await conditionsByRuleId()
const rows = db.select().from(alertRules).all() const rows = await db.select().from(alertRules)
return rows.map((r) => { return rows.map((r) => {
const targets = tgtMap.get(r.id) const targets = tgtMap.get(r.id)
const ts = targets?.length ? targets : [r.target] const ts = targets?.length ? targets : [r.target]
@@ -207,23 +202,23 @@ export type ReplaceAlertGroupInput = {
} }
/** Атомарно заменяет группы, правила и строки targets. */ /** Атомарно заменяет группы, правила и строки targets. */
export function replaceAlertsConfig(payload: { groups: ReplaceAlertGroupInput[]; rules: ReplaceAlertRuleInput[] }) { export async function replaceAlertsConfig(payload: { groups: ReplaceAlertGroupInput[]; rules: ReplaceAlertRuleInput[] }) {
const now = sql`(datetime('now'))` const now = sql`now()`
const { groups, rules } = payload const { groups, rules } = payload
db.transaction((tx) => { await db.transaction(async (tx) => {
tx.delete(alertRuleConditions).run() await tx.delete(alertRuleConditions)
tx.delete(alertRuleTargets).run() await tx.delete(alertRuleTargets)
tx.delete(alertRules).run() await tx.delete(alertRules)
tx.delete(alertGroups).run() await tx.delete(alertGroups)
for (const g of groups) { for (const g of groups) {
tx.insert(alertGroups).values({ await tx.insert(alertGroups).values({
id: g.id, id: g.id,
name: g.name.trim() || g.id, name: g.name.trim() || g.id,
combineMode: g.combineMode, combineMode: g.combineMode,
enabled: g.enabled, enabled: g.enabled,
cooldownOverride: g.cooldownOverride ?? null, cooldownOverride: g.cooldownOverride ?? null,
updatedAt: now, updatedAt: now,
}).run() })
} }
for (const r of rules) { for (const r of rules) {
const rawTargets = const rawTargets =
@@ -250,7 +245,7 @@ export function replaceAlertsConfig(payload: { groups: ReplaceAlertGroupInput[];
r.recoveryStabilitySec > 0 r.recoveryStabilitySec > 0
? Math.min(86400, Math.max(1, Math.floor(r.recoveryStabilitySec))) ? Math.min(86400, Math.max(1, Math.floor(r.recoveryStabilitySec)))
: null : null
tx.insert(alertRules).values({ await tx.insert(alertRules).values({
id: r.id, id: r.id,
name: r.name, name: r.name,
type: r.type, type: r.type,
@@ -265,39 +260,39 @@ export function replaceAlertsConfig(payload: { groups: ReplaceAlertGroupInput[];
chatId: r.chatId ?? "", chatId: r.chatId ?? "",
groupId: r.groupId && r.groupId.trim() ? r.groupId.trim() : null, groupId: r.groupId && r.groupId.trim() ? r.groupId.trim() : null,
updatedAt: now, updatedAt: now,
}).run() })
rawTargets.forEach((t, i) => { for (const [i, t] of rawTargets.entries()) {
tx.insert(alertRuleTargets).values({ await tx.insert(alertRuleTargets).values({
id: `rt-${r.id}-${i}`, id: `rt-${r.id}-${i}`,
ruleId: r.id, ruleId: r.id,
target: t, target: t,
sortIndex: i, sortIndex: i,
}).run() })
}) }
rawConds.forEach((c, i) => { for (const [i, c] of rawConds.entries()) {
tx.insert(alertRuleConditions).values({ await tx.insert(alertRuleConditions).values({
id: `rc-${r.id}-${i}`, id: `rc-${r.id}-${i}`,
ruleId: r.id, ruleId: r.id,
conditionLine: c, conditionLine: c,
sortIndex: i, sortIndex: i,
}).run() })
}) }
} }
}) })
} }
/** Совместимость: только правила, группы не трогаем. */ /** Совместимость: только правила, группы не трогаем. */
export function replaceAlertRules(rules: ReplaceAlertRuleInput[]) { export async function replaceAlertRules(rules: ReplaceAlertRuleInput[]) {
const curGroups = listAlertGroups() const curGroups = await listAlertGroups()
replaceAlertsConfig({ groups: curGroups.map((g) => ({ ...g })), rules }) await replaceAlertsConfig({ groups: curGroups.map((g) => ({ ...g })), rules })
} }
export function getTelegramSettingsRow() { export async function getTelegramSettingsRow() {
return ensureTelegramRow() return await ensureTelegramRow()
} }
export function getTelegramPublic() { export async function getTelegramPublic() {
const row = ensureTelegramRow() const row = await ensureTelegramRow()
const tid = row.messageThreadId const tid = row.messageThreadId
const messageThreadId = const messageThreadId =
typeof tid === "number" && Number.isFinite(tid) && Math.floor(tid) >= 1 ? Math.floor(tid) : null typeof tid === "number" && Number.isFinite(tid) && Math.floor(tid) >= 1 ? Math.floor(tid) : null
@@ -308,13 +303,13 @@ export function getTelegramPublic() {
} }
} }
export function updateTelegramSettings(patch: { export async function updateTelegramSettings(patch: {
token?: string | null token?: string | null
chatId?: string | undefined chatId?: string | undefined
/** undefined — не менять; null — сбросить тему */ /** undefined — не менять; null — сбросить тему */
messageThreadId?: number | null messageThreadId?: number | null
}) { }) {
const cur = ensureTelegramRow() const cur = await ensureTelegramRow()
let nextToken = cur.botToken let nextToken = cur.botToken
let nextChat = cur.chatId ?? "" let nextChat = cur.chatId ?? ""
let nextThread: number | null | undefined = undefined let nextThread: number | null | undefined = undefined
@@ -335,27 +330,26 @@ export function updateTelegramSettings(patch: {
const base = { const base = {
botToken: nextToken, botToken: nextToken,
chatId: nextChat, chatId: nextChat,
updatedAt: sql`(datetime('now'))`, updatedAt: sql`now()`,
} }
if (nextThread !== undefined) { if (nextThread !== undefined) {
db.update(alertTelegramSettings) await db.update(alertTelegramSettings)
.set({ ...base, messageThreadId: nextThread }) .set({ ...base, messageThreadId: nextThread })
.where(eq(alertTelegramSettings.id, 1)) .where(eq(alertTelegramSettings.id, 1))
.run()
} else { } else {
db.update(alertTelegramSettings).set(base).where(eq(alertTelegramSettings.id, 1)).run() await db.update(alertTelegramSettings).set(base).where(eq(alertTelegramSettings.id, 1))
} }
return getTelegramPublic() return await getTelegramPublic()
} }
/** Токен только для внутреннего вызова Telegram API (не отдаётся клиенту). */ /** Токен только для внутреннего вызова Telegram API (не отдаётся клиенту). */
export function getTelegramBotToken(): string { export async function getTelegramBotToken(): Promise<string> {
return ensureTelegramRow().botToken?.trim() ?? "" return (await ensureTelegramRow()).botToken?.trim() ?? ""
} }
/** Для `sendMessage`: только если в БД задана валидная тема. */ /** Для `sendMessage`: только если в БД задана валидная тема. */
export function getTelegramMessageThreadIdForApi(): number | undefined { export async function getTelegramMessageThreadIdForApi(): Promise<number | undefined> {
const { messageThreadId } = getTelegramPublic() const { messageThreadId } = await getTelegramPublic()
return messageThreadId ?? undefined return messageThreadId ?? undefined
} }
@@ -390,9 +384,9 @@ export async function sendTelegramAlertMessage(opts: {
/** Переопределение темы; undefined — как в настройках */ /** Переопределение темы; undefined — как в настройках */
messageThreadId?: number | null messageThreadId?: number | null
}): Promise<{ ok: true } | { ok: false; error: string }> { }): Promise<{ ok: true } | { ok: false; error: string }> {
const token = (opts.token?.trim() || getTelegramBotToken()).trim() const token = (opts.token?.trim() || await getTelegramBotToken()).trim()
if (!token) return { ok: false, error: "Не задан Bot Token" } if (!token) return { ok: false, error: "Не задан Bot Token" }
const pub = getTelegramPublic() const pub = await getTelegramPublic()
const chatId = (opts.chatId?.trim() || pub.chatId || "").trim() const chatId = (opts.chatId?.trim() || pub.chatId || "").trim()
if (!chatId) return { ok: false, error: "Не задан Chat ID" } if (!chatId) return { ok: false, error: "Не задан Chat ID" }
const threadId = const threadId =
@@ -400,7 +394,7 @@ export async function sendTelegramAlertMessage(opts: {
? opts.messageThreadId >= 1 ? opts.messageThreadId >= 1
? Math.floor(opts.messageThreadId) ? Math.floor(opts.messageThreadId)
: undefined : undefined
: getTelegramMessageThreadIdForApi() : await getTelegramMessageThreadIdForApi()
const url = `https://api.telegram.org/bot${encodeURIComponent(token)}/sendMessage` const url = `https://api.telegram.org/bot${encodeURIComponent(token)}/sendMessage`
try { try {
const payload: { chat_id: string; text: string; message_thread_id?: number } = { const payload: { chat_id: string; text: string; message_thread_id?: number } = {
@@ -423,7 +417,7 @@ export async function sendTelegramAlertMessage(opts: {
} }
} }
export function appendAlertHistory(entry: { export async function appendAlertHistory(entry: {
id: string id: string
ruleId?: string | null ruleId?: string | null
groupId?: string | null groupId?: string | null
@@ -434,7 +428,7 @@ export function appendAlertHistory(entry: {
firedAt?: string firedAt?: string
}) { }) {
const firedAt = entry.firedAt ?? new Date().toISOString() const firedAt = entry.firedAt ?? new Date().toISOString()
db.insert(alertHistory) await db.insert(alertHistory)
.values({ .values({
id: entry.id, id: entry.id,
ruleId: entry.ruleId ?? null, ruleId: entry.ruleId ?? null,
@@ -445,7 +439,6 @@ export function appendAlertHistory(entry: {
sentOk: entry.sentOk, sentOk: entry.sentOk,
firedAt, firedAt,
}) })
.run()
} }
interface LiveResRow { interface LiveResRow {
@@ -462,38 +455,38 @@ interface LiveProbeRow {
lossPct: number | null lossPct: number | null
} }
function loadLiveResourceIssues(): LiveResRow[] { async function loadLiveResourceIssues(): Promise<LiveResRow[]> {
return sqliteDatabase return dbAll<LiveResRow>(
.prepare( `SELECT s.name AS "serverName", urs.sampled_at AS "sampledAt"
`SELECT s.name AS serverName, urs.sampled_at AS sampledAt FROM uptime_resource_samples urs
FROM uptime_resource_samples urs INNER JOIN (
INNER JOIN ( SELECT DISTINCT ON (server_id) server_id, sampled_at, status
SELECT server_id, MAX(id) AS mid FROM uptime_resource_samples GROUP BY server_id FROM uptime_resource_samples
) latest ON latest.mid = urs.id ORDER BY server_id, sampled_at DESC
INNER JOIN servers s ON s.id = urs.server_id ) latest ON latest.server_id = urs.server_id AND latest.sampled_at = urs.sampled_at
WHERE urs.status = 'offline'`, INNER JOIN servers s ON s.id = urs.server_id
) WHERE urs.status = 'offline'`,
.all() as LiveResRow[] )
} }
function loadLiveProbeIssues(): LiveProbeRow[] { async function loadLiveProbeIssues(): Promise<LiveProbeRow[]> {
return sqliteDatabase return dbAll<LiveProbeRow>(
.prepare( `SELECT p.name AS "probeName", p.target AS target, ups.sampled_at AS "sampledAt",
`SELECT p.name AS probeName, p.target AS target, ups.sampled_at AS sampledAt, ups.status AS status, ups.rtt_ms AS "rttMs", ups.loss_pct AS "lossPct"
ups.status AS status, ups.rtt_ms AS rttMs, ups.loss_pct AS lossPct FROM uptime_probe_samples ups
FROM uptime_probe_samples ups INNER JOIN (
INNER JOIN ( SELECT DISTINCT ON (probe_id) probe_id, sampled_at, status, rtt_ms, loss_pct
SELECT probe_id, MAX(id) AS mid FROM uptime_probe_samples GROUP BY probe_id FROM uptime_probe_samples
) latest ON latest.mid = ups.id ORDER BY probe_id, sampled_at DESC
INNER JOIN uptime_probes p ON p.id = ups.probe_id ) latest ON latest.probe_id = ups.probe_id AND latest.sampled_at = ups.sampled_at
WHERE ups.status IN ('down', 'warn')`, INNER JOIN uptime_probes p ON p.id = ups.probe_id
) WHERE ups.status IN ('down', 'warn')`,
.all() as LiveProbeRow[] )
} }
function toLiveHistoryEntries(): ApiAlertHistoryEntry[] { async function toLiveHistoryEntries(): Promise<ApiAlertHistoryEntry[]> {
const out: ApiAlertHistoryEntry[] = [] const out: ApiAlertHistoryEntry[] = []
for (const r of loadLiveResourceIssues()) { for (const r of await loadLiveResourceIssues()) {
const safeId = `live-res-${r.serverName}-${r.sampledAt}`.replace(/[^a-zA-Z0-9._-]+/g, "_") const safeId = `live-res-${r.serverName}-${r.sampledAt}`.replace(/[^a-zA-Z0-9._-]+/g, "_")
out.push({ out.push({
id: safeId, id: safeId,
@@ -505,7 +498,7 @@ function toLiveHistoryEntries(): ApiAlertHistoryEntry[] {
source: "live", source: "live",
}) })
} }
for (const p of loadLiveProbeIssues()) { for (const p of await loadLiveProbeIssues()) {
const sev = p.status === "down" ? "critical" : "warning" const sev = p.status === "down" ? "critical" : "warning"
const loss = p.lossPct != null ? `${p.lossPct}%` : "—" const loss = p.lossPct != null ? `${p.lossPct}%` : "—"
const rtt = p.rttMs != null ? `${p.rttMs} мс` : "—" const rtt = p.rttMs != null ? `${p.rttMs} мс` : "—"
@@ -523,13 +516,12 @@ function toLiveHistoryEntries(): ApiAlertHistoryEntry[] {
return out return out
} }
export function listPersistedHistory(limit: number): ApiAlertHistoryEntry[] { export async function listPersistedHistory(limit: number): Promise<ApiAlertHistoryEntry[]> {
const rows = db const rows = await db
.select() .select()
.from(alertHistory) .from(alertHistory)
.orderBy(desc(alertHistory.firedAt)) .orderBy(desc(alertHistory.firedAt))
.limit(limit) .limit(limit)
.all()
return rows.map((h) => ({ return rows.map((h) => ({
id: h.id, id: h.id,
ruleName: h.ruleName, ruleName: h.ruleName,
@@ -541,9 +533,9 @@ export function listPersistedHistory(limit: number): ApiAlertHistoryEntry[] {
})) }))
} }
export function listMergedHistory(persistLimit = 40, liveCap = 25): ApiAlertHistoryEntry[] { export async function listMergedHistory(persistLimit = 40, liveCap = 25): Promise<ApiAlertHistoryEntry[]> {
const persisted = listPersistedHistory(persistLimit) const persisted = await listPersistedHistory(persistLimit)
const live = toLiveHistoryEntries().slice(0, liveCap) const live = (await toLiveHistoryEntries()).slice(0, liveCap)
const merged = [...persisted, ...live].sort((a, b) => (a.firedAt < b.firedAt ? 1 : a.firedAt > b.firedAt ? -1 : 0)) const merged = [...persisted, ...live].sort((a, b) => (a.firedAt < b.firedAt ? 1 : a.firedAt > b.firedAt ? -1 : 0))
const seen = new Set<string>() const seen = new Set<string>()
const dedup: ApiAlertHistoryEntry[] = [] const dedup: ApiAlertHistoryEntry[] = []
@@ -556,8 +548,8 @@ export function listMergedHistory(persistLimit = 40, liveCap = 25): ApiAlertHist
return dedup return dedup
} }
export function getAlertsMeta() { export async function getAlertsMeta() {
const srvRows = db const srvRows = await db
.select({ .select({
name: servers.name, name: servers.name,
host: servers.host, host: servers.host,
@@ -566,7 +558,6 @@ export function getAlertsMeta() {
type: servers.type, type: servers.type,
}) })
.from(servers) .from(servers)
.all()
const labels = srvRows.map((s) => (s.name?.trim() ? s.name.trim() : s.host)) const labels = srvRows.map((s) => (s.name?.trim() ? s.name.trim() : s.host))
const serversDetail = srvRows.map((r) => ({ const serversDetail = srvRows.map((r) => ({
name: r.name?.trim() || "", name: r.name?.trim() || "",
@@ -575,7 +566,7 @@ export function getAlertsMeta() {
country: r.country?.trim() || "", country: r.country?.trim() || "",
type: (r.type ?? "home-router") as string, type: (r.type ?? "home-router") as string,
})) }))
const probes = db.select({ name: uptimeProbes.name, target: uptimeProbes.target }).from(uptimeProbes).all() const probes = await db.select({ name: uptimeProbes.name, target: uptimeProbes.target }).from(uptimeProbes)
const probeTargets = probes.map((p) => `${p.name}${p.target}`) const probeTargets = probes.map((p) => `${p.name}${p.target}`)
const defaults = ["8.8.8.8", "1.1.1.1"] const defaults = ["8.8.8.8", "1.1.1.1"]
const rttLossTargets: string[] = [] const rttLossTargets: string[] = []
@@ -18,7 +18,7 @@ export function getBackupSchedulerCollectorState(): { running: boolean } {
export async function collectScheduledBackupsOnce(): Promise<BackupsRunSnapshot> { export async function collectScheduledBackupsOnce(): Promise<BackupsRunSnapshot> {
const sampledAt = new Date().toISOString() const sampledAt = new Date().toISOString()
const settings = getBackupScheduleSettings() const settings = await getBackupScheduleSettings()
const snapshot: BackupsRunSnapshot = { const snapshot: BackupsRunSnapshot = {
v: SCHEDULER_RUN_SNAPSHOT_VERSION, v: SCHEDULER_RUN_SNAPSHOT_VERSION,
job: "backups", job: "backups",
@@ -49,7 +49,7 @@ export async function collectScheduledBackupsOnce(): Promise<BackupsRunSnapshot>
if (settings.format !== "rsc") { if (settings.format !== "rsc") {
snapshot.skipped = true snapshot.skipped = true
snapshot.errors = ["Формат backup пока не поддерживается, используйте rsc"] snapshot.errors = ["Формат backup пока не поддерживается, используйте rsc"]
touchBackupScheduleRunMeta({ await touchBackupScheduleRunMeta({
lastRunAt: sampledAt, lastRunAt: sampledAt,
lastDurationMs: 0, lastDurationMs: 0,
lastError: snapshot.errors[0], lastError: snapshot.errors[0],
@@ -59,9 +59,9 @@ export async function collectScheduledBackupsOnce(): Promise<BackupsRunSnapshot>
collecting = true collecting = true
const started = Date.now() const started = Date.now()
const serverIds = resolveBackupServerIds(settings) const serverIds = await resolveBackupServerIds(settings)
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "backups.job.started", eventType: "backups.job.started",
sourceModule: "backups", sourceModule: "backups",
@@ -88,13 +88,13 @@ export async function collectScheduledBackupsOnce(): Promise<BackupsRunSnapshot>
snapshot.pruned += await pruneBackupsForServer(id, settings.keepCount) snapshot.pruned += await pruneBackupsForServer(id, settings.keepCount)
} }
touchBackupScheduleRunMeta({ await touchBackupScheduleRunMeta({
lastRunAt: sampledAt, lastRunAt: sampledAt,
lastDurationMs: Date.now() - started, lastDurationMs: Date.now() - started,
lastError: snapshot.errors?.length ? snapshot.errors.join("; ") : null, lastError: snapshot.errors?.length ? snapshot.errors.join("; ") : null,
}) })
appendEvent({ await appendEvent({
level: snapshot.failures > 0 ? "warning" : "info", level: snapshot.failures > 0 ? "warning" : "info",
eventType: "backups.job.done", eventType: "backups.job.done",
sourceModule: "backups", sourceModule: "backups",
@@ -115,12 +115,12 @@ export async function collectScheduledBackupsOnce(): Promise<BackupsRunSnapshot>
const message = e instanceof Error ? e.message : String(e) const message = e instanceof Error ? e.message : String(e)
snapshot.fatalError = message snapshot.fatalError = message
snapshot.errors?.push(message) snapshot.errors?.push(message)
touchBackupScheduleRunMeta({ await touchBackupScheduleRunMeta({
lastRunAt: sampledAt, lastRunAt: sampledAt,
lastDurationMs: Date.now() - started, lastDurationMs: Date.now() - started,
lastError: message, lastError: message,
}) })
appendEvent({ await appendEvent({
level: "critical", level: "critical",
eventType: "backups.job.failed", eventType: "backups.job.failed",
sourceModule: "backups", sourceModule: "backups",
+38 -43
View File
@@ -4,6 +4,7 @@ import path from "node:path"
import { desc, eq } from "drizzle-orm" import { desc, eq } from "drizzle-orm"
import type { BackupScheduleSettingsDto } from "@mmapp/contracts/backups" import type { BackupScheduleSettingsDto } from "@mmapp/contracts/backups"
import { db } from "../db/index.js" import { db } from "../db/index.js"
import { parseJsonArray } from "../db/json.js"
import { backupEntries, backupScheduleSettings } from "../db/schema.js" import { backupEntries, backupScheduleSettings } from "../db/schema.js"
import { getServerRowById } from "../modules/servers/repository/servers-repository.js" import { getServerRowById } from "../modules/servers/repository/servers-repository.js"
import { listServersRead } from "../modules/servers/service/servers-service.js" import { listServersRead } from "../modules/servers/service/servers-service.js"
@@ -14,7 +15,7 @@ const BACKUPS_DIR = path.resolve(process.cwd(), "storage", "backups")
export type BackupMeta = { export type BackupMeta = {
id: string id: string
serverId: string serverId: number | null
serverName: string serverName: string
filename: string filename: string
sizeBytes: number sizeBytes: number
@@ -40,17 +41,18 @@ export async function ensureBackupStorage(): Promise<void> {
await mkdir(BACKUPS_DIR, { recursive: true }) await mkdir(BACKUPS_DIR, { recursive: true })
} }
export function listBackups(): BackupMeta[] { export async function listBackups(): Promise<BackupMeta[]> {
return db.select().from(backupEntries).orderBy(desc(backupEntries.createdAt)).all().map(rowToMeta) const rows = await db.select().from(backupEntries).orderBy(desc(backupEntries.createdAt))
return rows.map(rowToMeta)
} }
export function getBackupById(id: string): BackupMeta | null { export async function getBackupById(id: string): Promise<BackupMeta | null> {
const row = db.select().from(backupEntries).where(eq(backupEntries.id, id)).limit(1).all()[0] const row = (await db.select().from(backupEntries).where(eq(backupEntries.id, id)).limit(1))[0]
return row ? rowToMeta(row) : null return row ? rowToMeta(row) : null
} }
export function insertBackup(meta: BackupMeta): void { export async function insertBackup(meta: BackupMeta): Promise<void> {
db.insert(backupEntries).values({ await db.insert(backupEntries).values({
id: meta.id, id: meta.id,
serverId: meta.serverId, serverId: meta.serverId,
serverName: meta.serverName, serverName: meta.serverName,
@@ -59,13 +61,13 @@ export function insertBackup(meta: BackupMeta): void {
kind: meta.kind, kind: meta.kind,
notes: meta.notes ?? null, notes: meta.notes ?? null,
createdAt: meta.createdAt, createdAt: meta.createdAt,
}).run() })
} }
export async function deleteBackupRecord(id: string): Promise<BackupMeta | null> { export async function deleteBackupRecord(id: string): Promise<BackupMeta | null> {
const hit = getBackupById(id) const hit = await getBackupById(id)
if (!hit) return null if (!hit) return null
db.delete(backupEntries).where(eq(backupEntries.id, id)).run() await db.delete(backupEntries).where(eq(backupEntries.id, id))
await rm(path.join(BACKUPS_DIR, hit.filename), { force: true }) await rm(path.join(BACKUPS_DIR, hit.filename), { force: true })
return hit return hit
} }
@@ -75,18 +77,12 @@ function fmtTs(d = new Date()): string {
return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())}_${p(d.getHours())}-${p(d.getMinutes())}-${p(d.getSeconds())}` return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())}_${p(d.getHours())}-${p(d.getMinutes())}-${p(d.getSeconds())}`
} }
function parseServerIds(raw: string): string[] { function parseServerIds(raw: unknown): string[] {
try { return parseJsonArray(raw).map(String).filter(Boolean)
const parsed = JSON.parse(raw) as unknown
if (!Array.isArray(parsed)) return []
return parsed.map(String).filter(Boolean)
} catch {
return []
}
} }
function getBackupScheduleSettingsRow() { async function getBackupScheduleSettingsRow() {
return db.select().from(backupScheduleSettings).where(eq(backupScheduleSettings.id, SETTINGS_ID)).limit(1).all()[0] return (await db.select().from(backupScheduleSettings).where(eq(backupScheduleSettings.id, SETTINGS_ID)).limit(1))[0]
?? { ?? {
id: SETTINGS_ID, id: SETTINGS_ID,
enabled: true, enabled: true,
@@ -97,7 +93,7 @@ function getBackupScheduleSettingsRow() {
monthDay: 1, monthDay: 1,
keepCount: 7, keepCount: 7,
format: "rsc" as const, format: "rsc" as const,
serverIdsJson: "[]", serverIdsJson: [],
lastRunAt: null, lastRunAt: null,
lastDurationMs: null, lastDurationMs: null,
lastError: null, lastError: null,
@@ -105,8 +101,8 @@ function getBackupScheduleSettingsRow() {
} }
} }
export function getBackupScheduleSettings(): BackupScheduleSettingsDto { export async function getBackupScheduleSettings(): Promise<BackupScheduleSettingsDto> {
const row = getBackupScheduleSettingsRow() const row = await getBackupScheduleSettingsRow()
return { return {
enabled: row.enabled, enabled: row.enabled,
frequency: row.frequency, frequency: row.frequency,
@@ -124,7 +120,7 @@ export function getBackupScheduleSettings(): BackupScheduleSettingsDto {
} }
} }
export function updateBackupScheduleSettings(patch: Partial<{ export async function updateBackupScheduleSettings(patch: Partial<{
enabled: boolean enabled: boolean
frequency: "daily" | "weekly" | "monthly" frequency: "daily" | "weekly" | "monthly"
hour: number hour: number
@@ -135,7 +131,7 @@ export function updateBackupScheduleSettings(patch: Partial<{
format: "rsc" | "backup" format: "rsc" | "backup"
serverIds: string[] serverIds: string[]
}>) { }>) {
const prev = getBackupScheduleSettingsRow() const prev = await getBackupScheduleSettingsRow()
const now = new Date().toISOString() const now = new Date().toISOString()
const next = { const next = {
enabled: patch.enabled ?? prev.enabled, enabled: patch.enabled ?? prev.enabled,
@@ -146,33 +142,33 @@ export function updateBackupScheduleSettings(patch: Partial<{
monthDay: patch.monthDay ?? prev.monthDay, monthDay: patch.monthDay ?? prev.monthDay,
keepCount: patch.keepCount ?? prev.keepCount, keepCount: patch.keepCount ?? prev.keepCount,
format: patch.format ?? prev.format, format: patch.format ?? prev.format,
serverIdsJson: patch.serverIds ? JSON.stringify(patch.serverIds) : prev.serverIdsJson, serverIdsJson: patch.serverIds ?? prev.serverIdsJson,
updatedAt: now, updatedAt: now,
} }
if (db.select().from(backupScheduleSettings).where(eq(backupScheduleSettings.id, SETTINGS_ID)).limit(1).all()[0]) { if ((await db.select().from(backupScheduleSettings).where(eq(backupScheduleSettings.id, SETTINGS_ID)).limit(1))[0]) {
db.update(backupScheduleSettings).set(next).where(eq(backupScheduleSettings.id, SETTINGS_ID)).run() await db.update(backupScheduleSettings).set(next).where(eq(backupScheduleSettings.id, SETTINGS_ID))
} else { } else {
db.insert(backupScheduleSettings).values({ id: SETTINGS_ID, ...next }).run() await db.insert(backupScheduleSettings).values({ id: SETTINGS_ID, ...next })
} }
return getBackupScheduleSettings() return await getBackupScheduleSettings()
} }
export function touchBackupScheduleRunMeta(patch: { export async function touchBackupScheduleRunMeta(patch: {
lastRunAt?: string lastRunAt?: string
lastDurationMs?: number lastDurationMs?: number
lastError?: string | null lastError?: string | null
}) { }) {
const prev = getBackupScheduleSettingsRow() const prev = await getBackupScheduleSettingsRow()
db.update(backupScheduleSettings).set({ await db.update(backupScheduleSettings).set({
lastRunAt: patch.lastRunAt ?? prev.lastRunAt, lastRunAt: patch.lastRunAt ?? prev.lastRunAt,
lastDurationMs: patch.lastDurationMs ?? prev.lastDurationMs, lastDurationMs: patch.lastDurationMs ?? prev.lastDurationMs,
lastError: patch.lastError === undefined ? prev.lastError : patch.lastError, lastError: patch.lastError === undefined ? prev.lastError : patch.lastError,
updatedAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
}).where(eq(backupScheduleSettings.id, SETTINGS_ID)).run() }).where(eq(backupScheduleSettings.id, SETTINGS_ID))
} }
export function resolveBackupServerIds(settings: BackupScheduleSettingsDto): string[] { export async function resolveBackupServerIds(settings: BackupScheduleSettingsDto): Promise<string[]> {
const enabled = new Set(listServersRead().map((s) => String(s.id))) const enabled = new Set((await listServersRead()).map((s) => String(s.id)))
const requested = settings.serverIds.length > 0 ? settings.serverIds : [...enabled] const requested = settings.serverIds.length > 0 ? settings.serverIds : [...enabled]
return [...new Set(requested)].filter((id) => enabled.has(id)) return [...new Set(requested)].filter((id) => enabled.has(id))
} }
@@ -227,7 +223,7 @@ export async function runBackupForServer(
if (!Number.isFinite(serverIdNum)) { if (!Number.isFinite(serverIdNum)) {
throw new Error("Невалидный id сервера") throw new Error("Невалидный id сервера")
} }
const row = getServerRowById(serverIdNum) const row = await getServerRowById(serverIdNum)
if (!row) { if (!row) {
throw new Error("Сервер не найден") throw new Error("Сервер не найден")
} }
@@ -242,7 +238,7 @@ export async function runBackupForServer(
const st = await stat(filePath) const st = await stat(filePath)
const meta: BackupMeta = { const meta: BackupMeta = {
id: randomUUID(), id: randomUUID(),
serverId: String(row.id), serverId: row.id,
serverName: row.name, serverName: row.name,
filename, filename,
sizeBytes: st.size, sizeBytes: st.size,
@@ -250,19 +246,18 @@ export async function runBackupForServer(
kind, kind,
notes, notes,
} }
insertBackup(meta) await insertBackup(meta)
return meta return meta
} }
export async function pruneBackupsForServer(serverId: string, keepCount: number): Promise<number> { export async function pruneBackupsForServer(serverId: string, keepCount: number): Promise<number> {
const rows = db.select().from(backupEntries) const rows = await db.select().from(backupEntries)
.where(eq(backupEntries.serverId, serverId)) .where(eq(backupEntries.serverId, Number(serverId)))
.orderBy(desc(backupEntries.createdAt)) .orderBy(desc(backupEntries.createdAt))
.all()
if (rows.length <= keepCount) return 0 if (rows.length <= keepCount) return 0
const toDelete = rows.slice(keepCount) const toDelete = rows.slice(keepCount)
for (const hit of toDelete) { for (const hit of toDelete) {
db.delete(backupEntries).where(eq(backupEntries.id, hit.id)).run() await db.delete(backupEntries).where(eq(backupEntries.id, hit.id))
await rm(path.join(BACKUPS_DIR, hit.filename), { force: true }) await rm(path.join(BACKUPS_DIR, hit.filename), { force: true })
} }
return toDelete.length return toDelete.length
+1 -1
View File
@@ -14,7 +14,7 @@ export function bgpPeerAlertKey(s: BgpSessionRead): string {
/** Агрегат BGP-сессий с включённых серверов — как `/api/bgp/sessions`. */ /** Агрегат BGP-сессий с включённых серверов — как `/api/bgp/sessions`. */
export async function fetchBgpSessionsForAlerts(): Promise<BgpSessionRead[]> { export async function fetchBgpSessionsForAlerts(): Promise<BgpSessionRead[]> {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const results = await Promise.all( const results = await Promise.all(
allServers.map(async (server) => { allServers.map(async (server) => {
try { try {
@@ -1,3 +1,4 @@
import { parseJsonArray } from "../db/json.js"
import { appendEvent } from "../modules/events/service/events-service.js" import { appendEvent } from "../modules/events/service/events-service.js"
import { issueCertificateWithCloudflareDns } from "./acme-cloudflare.js" import { issueCertificateWithCloudflareDns } from "./acme-cloudflare.js"
import { import {
@@ -11,15 +12,15 @@ const runningJobs = new Set<string>()
export async function runCertificateIssueJob(jobId: string): Promise<void> { export async function runCertificateIssueJob(jobId: string): Promise<void> {
if (runningJobs.has(jobId)) return if (runningJobs.has(jobId)) return
runningJobs.add(jobId) runningJobs.add(jobId)
const row = getIssueJobRecord(jobId) const row = await getIssueJobRecord(jobId)
if (!row) { if (!row) {
runningJobs.delete(jobId) runningJobs.delete(jobId)
return return
} }
const server = getServerRowByIdString(row.serverId) const server = await getServerRowByIdString(row.serverId)
if (!server) { if (!server) {
updateIssueJobRecord(jobId, { await updateIssueJobRecord(jobId, {
status: "failed", status: "failed",
step: "failed", step: "failed",
finishedAt: new Date().toISOString(), finishedAt: new Date().toISOString(),
@@ -29,20 +30,15 @@ export async function runCertificateIssueJob(jobId: string): Promise<void> {
return return
} }
let domainNames: string[] = [] const domainNames = parseJsonArray(row.domainNames).map(String)
try {
domainNames = JSON.parse(row.domainNames) as string[]
} catch {
domainNames = []
}
const trustStore = row.trustStore.split(",").map((s) => s.trim()).filter(Boolean) const trustStore = row.trustStore.split(",").map((s) => s.trim()).filter(Boolean)
const startedAt = new Date().toISOString() const startedAt = new Date().toISOString()
const isScheduler = row.source === "scheduler" const isScheduler = row.source === "scheduler"
updateIssueJobRecord(jobId, { status: "running", step: "acme_order", startedAt, error: null }) await updateIssueJobRecord(jobId, { status: "running", step: "acme_order", startedAt, error: null })
if (isScheduler) { if (isScheduler) {
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "certificates.renew.started", eventType: "certificates.renew.started",
sourceModule: "certificates", sourceModule: "certificates",
@@ -60,15 +56,15 @@ export async function runCertificateIssueJob(jobId: string): Promise<void> {
domainNames, domainNames,
keyType: row.keyType === "ec256" ? "ec256" : "rsa2048", keyType: row.keyType === "ec256" ? "ec256" : "rsa2048",
trustStore: trustStore.length > 0 ? trustStore : ["www", "api"], trustStore: trustStore.length > 0 ? trustStore : ["www", "api"],
onStep: (step) => updateIssueJobRecord(jobId, { step }), onStep: (step) => { void updateIssueJobRecord(jobId, { step }) },
}) })
updateIssueJobRecord(jobId, { await updateIssueJobRecord(jobId, {
status: "done", status: "done",
step: "done", step: "done",
finishedAt: new Date().toISOString(), finishedAt: new Date().toISOString(),
error: null, error: null,
}) })
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: isScheduler ? "certificates.renew.done" : "certificates.issue.done", eventType: isScheduler ? "certificates.renew.done" : "certificates.issue.done",
sourceModule: "certificates", sourceModule: "certificates",
@@ -79,13 +75,13 @@ export async function runCertificateIssueJob(jobId: string): Promise<void> {
}) })
} catch (e) { } catch (e) {
const message = e instanceof Error ? e.message : String(e) const message = e instanceof Error ? e.message : String(e)
updateIssueJobRecord(jobId, { await updateIssueJobRecord(jobId, {
status: "failed", status: "failed",
step: "failed", step: "failed",
finishedAt: new Date().toISOString(), finishedAt: new Date().toISOString(),
error: message, error: message,
}) })
appendEvent({ await appendEvent({
level: "warning", level: "warning",
eventType: isScheduler ? "certificates.renew.failed" : "certificates.issue.failed", eventType: isScheduler ? "certificates.renew.failed" : "certificates.issue.failed",
sourceModule: "certificates", sourceModule: "certificates",
@@ -22,7 +22,7 @@ export function getCertificateRenewCollectorState(): { running: boolean } {
export async function collectCertificatesRenewOnce(): Promise<CertificatesRenewRunSnapshot> { export async function collectCertificatesRenewOnce(): Promise<CertificatesRenewRunSnapshot> {
const sampledAt = new Date().toISOString() const sampledAt = new Date().toISOString()
const settings = getCertificateRenewSettings() const settings = await getCertificateRenewSettings()
if (collecting) { if (collecting) {
return { return {
v: SCHEDULER_RUN_SNAPSHOT_VERSION, v: SCHEDULER_RUN_SNAPSHOT_VERSION,
@@ -55,7 +55,7 @@ export async function collectCertificatesRenewOnce(): Promise<CertificatesRenewR
return snapshot return snapshot
} }
const targets = listManagedCertificateTargets() const targets = await listManagedCertificateTargets()
for (const target of targets) { for (const target of targets) {
snapshot.checked += 1 snapshot.checked += 1
const item = { const item = {
@@ -66,12 +66,12 @@ export async function collectCertificatesRenewOnce(): Promise<CertificatesRenewR
message: "", message: "",
} }
if (hasActiveIssueJob(target.serverId, target.certName)) { if (await hasActiveIssueJob(target.serverId, target.certName)) {
item.action = "skipped" item.action = "skipped"
item.message = "Уже выполняется выпуск" item.message = "Уже выполняется выпуск"
snapshot.skippedTargets += 1 snapshot.skippedTargets += 1
snapshot.targets?.push(item) snapshot.targets?.push(item)
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "certificates.renew.skipped", eventType: "certificates.renew.skipped",
sourceModule: "certificates", sourceModule: "certificates",
@@ -83,7 +83,7 @@ export async function collectCertificatesRenewOnce(): Promise<CertificatesRenewR
continue continue
} }
const server = getServerRowByIdString(target.serverId) const server = await getServerRowByIdString(target.serverId)
if (!server) { if (!server) {
item.action = "error" item.action = "error"
item.message = "Сервер не найден" item.message = "Сервер не найден"
@@ -114,7 +114,7 @@ export async function collectCertificatesRenewOnce(): Promise<CertificatesRenewR
} }
const jobId = randomUUID() const jobId = randomUUID()
createIssueJobRecord({ await createIssueJobRecord({
id: jobId, id: jobId,
serverId: target.serverId, serverId: target.serverId,
certName: target.certName, certName: target.certName,
@@ -137,7 +137,7 @@ export async function collectCertificatesRenewOnce(): Promise<CertificatesRenewR
} }
} }
touchCertificateRenewRunMeta({ await touchCertificateRenewRunMeta({
lastCollectedAt: sampledAt, lastCollectedAt: sampledAt,
lastDurationMs: Date.now() - started, lastDurationMs: Date.now() - started,
lastError: snapshot.errors.length > 0 ? snapshot.errors.join("; ") : null, lastError: snapshot.errors.length > 0 ? snapshot.errors.join("; ") : null,
@@ -146,7 +146,7 @@ export async function collectCertificatesRenewOnce(): Promise<CertificatesRenewR
} catch (e) { } catch (e) {
const message = e instanceof Error ? e.message : String(e) const message = e instanceof Error ? e.message : String(e)
snapshot.fatalError = message snapshot.fatalError = message
touchCertificateRenewRunMeta({ await touchCertificateRenewRunMeta({
lastCollectedAt: sampledAt, lastCollectedAt: sampledAt,
lastDurationMs: Date.now() - started, lastDurationMs: Date.now() - started,
lastError: message, lastError: message,
+51 -65
View File
@@ -1,6 +1,7 @@
import { desc, eq, inArray } from "drizzle-orm" import { desc, eq, inArray } from "drizzle-orm"
import type { CertificateDto, CertificateRenewSettingsDto } from "@mmapp/contracts/certificates" import type { CertificateDto, CertificateRenewSettingsDto } from "@mmapp/contracts/certificates"
import { db } from "../db/index.js" import { db } from "../db/index.js"
import { parseJsonArray } from "../db/json.js"
import { import {
acmeSettings, acmeSettings,
certificateIssueJobs, certificateIssueJobs,
@@ -25,7 +26,7 @@ export async function listCertificatesFromServers(): Promise<{
certificates: CertificateDto[] certificates: CertificateDto[]
failures: Array<{ serverId: string; serverName?: string; error: string }> failures: Array<{ serverId: string; serverName?: string; error: string }>
}> { }> {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const certificates: CertificateDto[] = [] const certificates: CertificateDto[] = []
const failures: Array<{ serverId: string; serverName?: string; error: string }> = [] const failures: Array<{ serverId: string; serverName?: string; error: string }> = []
@@ -53,8 +54,8 @@ export async function listCertificatesFromServers(): Promise<{
return { certificates, failures } return { certificates, failures }
} }
export function getAcmeSettingsPublic() { export async function getAcmeSettingsPublic() {
const row = db.select().from(acmeSettings).where(eq(acmeSettings.id, SETTINGS_ID)).limit(1).all()[0] const row = (await db.select().from(acmeSettings).where(eq(acmeSettings.id, SETTINGS_ID)).limit(1))[0]
if (!row) { if (!row) {
return { return {
directoryUrl: "https://acme-v02.api.letsencrypt.org/directory", directoryUrl: "https://acme-v02.api.letsencrypt.org/directory",
@@ -71,30 +72,29 @@ export function getAcmeSettingsPublic() {
} }
} }
export function getAcmeCloudflareToken(): string { export async function getAcmeCloudflareToken(): Promise<string> {
const row = db.select().from(acmeSettings).where(eq(acmeSettings.id, SETTINGS_ID)).limit(1).all()[0] const row = (await db.select().from(acmeSettings).where(eq(acmeSettings.id, SETTINGS_ID)).limit(1))[0]
return row?.cloudflareApiToken?.trim() ?? "" return row?.cloudflareApiToken?.trim() ?? ""
} }
export function getAcmeAccountPrivateKey(): string { export async function getAcmeAccountPrivateKey(): Promise<string> {
const row = db.select().from(acmeSettings).where(eq(acmeSettings.id, SETTINGS_ID)).limit(1).all()[0] const row = (await db.select().from(acmeSettings).where(eq(acmeSettings.id, SETTINGS_ID)).limit(1))[0]
return row?.accountPrivateKey?.trim() ?? "" return row?.accountPrivateKey?.trim() ?? ""
} }
export function saveAcmeAccountPrivateKey(pem: string) { export async function saveAcmeAccountPrivateKey(pem: string) {
const now = new Date().toISOString() const now = new Date().toISOString()
db.update(acmeSettings) await db.update(acmeSettings)
.set({ accountPrivateKey: pem, updatedAt: now }) .set({ accountPrivateKey: pem, updatedAt: now })
.where(eq(acmeSettings.id, SETTINGS_ID)) .where(eq(acmeSettings.id, SETTINGS_ID))
.run()
} }
export function updateAcmeSettings(input: { export async function updateAcmeSettings(input: {
directoryUrl?: string directoryUrl?: string
defaultZoneId?: string | null defaultZoneId?: string | null
cloudflareApiToken?: string | null cloudflareApiToken?: string | null
}) { }) {
const row = db.select().from(acmeSettings).where(eq(acmeSettings.id, SETTINGS_ID)).limit(1).all()[0] const row = (await db.select().from(acmeSettings).where(eq(acmeSettings.id, SETTINGS_ID)).limit(1))[0]
const now = new Date().toISOString() const now = new Date().toISOString()
const next = { const next = {
directoryUrl: input.directoryUrl?.trim() || row?.directoryUrl || "https://acme-v02.api.letsencrypt.org/directory", directoryUrl: input.directoryUrl?.trim() || row?.directoryUrl || "https://acme-v02.api.letsencrypt.org/directory",
@@ -109,14 +109,14 @@ export function updateAcmeSettings(input: {
updatedAt: now, updatedAt: now,
} }
if (row) { if (row) {
db.update(acmeSettings).set(next).where(eq(acmeSettings.id, SETTINGS_ID)).run() await db.update(acmeSettings).set(next).where(eq(acmeSettings.id, SETTINGS_ID))
} else { } else {
db.insert(acmeSettings).values({ id: SETTINGS_ID, accountPrivateKey: "", ...next }).run() await db.insert(acmeSettings).values({ id: SETTINGS_ID, accountPrivateKey: "", ...next })
} }
return getAcmeSettingsPublic() return await getAcmeSettingsPublic()
} }
export function createIssueJobRecord(input: { export async function createIssueJobRecord(input: {
id: string id: string
serverId: string serverId: string
certName: string certName: string
@@ -126,21 +126,21 @@ export function createIssueJobRecord(input: {
source?: "manual" | "scheduler" source?: "manual" | "scheduler"
}) { }) {
const now = new Date().toISOString() const now = new Date().toISOString()
db.insert(certificateIssueJobs).values({ await db.insert(certificateIssueJobs).values({
id: input.id, id: input.id,
status: "queued", status: "queued",
step: "queued", step: "queued",
source: input.source ?? "manual", source: input.source ?? "manual",
serverId: input.serverId, serverId: Number(input.serverId),
certName: input.certName, certName: input.certName,
domainNames: JSON.stringify(input.domainNames), domainNames: input.domainNames,
keyType: input.keyType, keyType: input.keyType,
trustStore: input.trustStore, trustStore: input.trustStore,
requestedAt: now, requestedAt: now,
}).run() })
} }
export function updateIssueJobRecord( export async function updateIssueJobRecord(
id: string, id: string,
patch: Partial<{ patch: Partial<{
status: "queued" | "running" | "done" | "failed" status: "queued" | "running" | "done" | "failed"
@@ -150,21 +150,15 @@ export function updateIssueJobRecord(
error: string | null error: string | null
}>, }>,
) { ) {
db.update(certificateIssueJobs).set(patch).where(eq(certificateIssueJobs.id, id)).run() await db.update(certificateIssueJobs).set(patch).where(eq(certificateIssueJobs.id, id))
} }
export function getIssueJobRecord(id: string) { export async function getIssueJobRecord(id: string) {
return db.select().from(certificateIssueJobs).where(eq(certificateIssueJobs.id, id)).limit(1).all()[0] ?? null return (await db.select().from(certificateIssueJobs).where(eq(certificateIssueJobs.id, id)).limit(1))[0] ?? null
} }
export function toIssueJobDto(row: NonNullable<ReturnType<typeof getIssueJobRecord>>) { export function toIssueJobDto(row: NonNullable<Awaited<ReturnType<typeof getIssueJobRecord>>>) {
let domainNames: string[] = [] const domainNames = parseJsonArray(row.domainNames).map(String)
try {
const parsed = JSON.parse(row.domainNames) as unknown
if (Array.isArray(parsed)) domainNames = parsed.map(String)
} catch {
domainNames = []
}
return { return {
id: row.id, id: row.id,
status: row.status, status: row.status,
@@ -179,14 +173,14 @@ export function toIssueJobDto(row: NonNullable<ReturnType<typeof getIssueJobReco
} }
} }
export function getServerRowByIdString(serverId: string) { export async function getServerRowByIdString(serverId: string | number) {
const id = Number.parseInt(serverId, 10) const id = typeof serverId === "number" ? serverId : Number.parseInt(serverId, 10)
if (!Number.isFinite(id)) return null if (!Number.isFinite(id)) return null
return db.select().from(servers).where(eq(servers.id, id)).limit(1).all()[0] ?? null return (await db.select().from(servers).where(eq(servers.id, id)).limit(1))[0] ?? null
} }
function getCertificateRenewSettingsRow() { async function getCertificateRenewSettingsRow() {
return db.select().from(certificateRenewSettings).where(eq(certificateRenewSettings.id, SETTINGS_ID)).limit(1).all()[0] return (await db.select().from(certificateRenewSettings).where(eq(certificateRenewSettings.id, SETTINGS_ID)).limit(1))[0]
?? { ?? {
id: SETTINGS_ID, id: SETTINGS_ID,
enabled: true, enabled: true,
@@ -199,8 +193,8 @@ function getCertificateRenewSettingsRow() {
} }
} }
export function getCertificateRenewSettings(): CertificateRenewSettingsDto { export async function getCertificateRenewSettings(): Promise<CertificateRenewSettingsDto> {
const row = getCertificateRenewSettingsRow() const row = await getCertificateRenewSettingsRow()
return { return {
enabled: row.enabled, enabled: row.enabled,
intervalSec: row.intervalSec, intervalSec: row.intervalSec,
@@ -212,12 +206,12 @@ export function getCertificateRenewSettings(): CertificateRenewSettingsDto {
} }
} }
export function updateCertificateRenewSettings(patch: { export async function updateCertificateRenewSettings(patch: {
enabled?: boolean enabled?: boolean
intervalSec?: number intervalSec?: number
renewBeforeDays?: number renewBeforeDays?: number
}) { }) {
const prev = getCertificateRenewSettingsRow() const prev = await getCertificateRenewSettingsRow()
const now = new Date().toISOString() const now = new Date().toISOString()
const next = { const next = {
enabled: patch.enabled ?? prev.enabled, enabled: patch.enabled ?? prev.enabled,
@@ -225,47 +219,40 @@ export function updateCertificateRenewSettings(patch: {
renewBeforeDays: patch.renewBeforeDays ?? prev.renewBeforeDays, renewBeforeDays: patch.renewBeforeDays ?? prev.renewBeforeDays,
updatedAt: now, updatedAt: now,
} }
if (db.select().from(certificateRenewSettings).where(eq(certificateRenewSettings.id, SETTINGS_ID)).limit(1).all()[0]) { if ((await db.select().from(certificateRenewSettings).where(eq(certificateRenewSettings.id, SETTINGS_ID)).limit(1))[0]) {
db.update(certificateRenewSettings).set(next).where(eq(certificateRenewSettings.id, SETTINGS_ID)).run() await db.update(certificateRenewSettings).set(next).where(eq(certificateRenewSettings.id, SETTINGS_ID))
} else { } else {
db.insert(certificateRenewSettings).values({ id: SETTINGS_ID, ...next }).run() await db.insert(certificateRenewSettings).values({ id: SETTINGS_ID, ...next })
} }
return getCertificateRenewSettings() return await getCertificateRenewSettings()
} }
export function touchCertificateRenewRunMeta(patch: { export async function touchCertificateRenewRunMeta(patch: {
lastCollectedAt?: string lastCollectedAt?: string
lastDurationMs?: number lastDurationMs?: number
lastError?: string | null lastError?: string | null
}) { }) {
const prev = getCertificateRenewSettingsRow() const prev = await getCertificateRenewSettingsRow()
db.update(certificateRenewSettings).set({ await db.update(certificateRenewSettings).set({
lastCollectedAt: patch.lastCollectedAt ?? prev.lastCollectedAt, lastCollectedAt: patch.lastCollectedAt ?? prev.lastCollectedAt,
lastDurationMs: patch.lastDurationMs ?? prev.lastDurationMs, lastDurationMs: patch.lastDurationMs ?? prev.lastDurationMs,
lastError: patch.lastError === undefined ? prev.lastError : patch.lastError, lastError: patch.lastError === undefined ? prev.lastError : patch.lastError,
updatedAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
}).where(eq(certificateRenewSettings.id, SETTINGS_ID)).run() }).where(eq(certificateRenewSettings.id, SETTINGS_ID))
} }
export function listManagedCertificateTargets(): ManagedCertificateTarget[] { export async function listManagedCertificateTargets(): Promise<ManagedCertificateTarget[]> {
const rows = db.select().from(certificateIssueJobs) const rows = await db.select().from(certificateIssueJobs)
.where(eq(certificateIssueJobs.status, "done")) .where(eq(certificateIssueJobs.status, "done"))
.orderBy(desc(certificateIssueJobs.finishedAt), desc(certificateIssueJobs.requestedAt)) .orderBy(desc(certificateIssueJobs.finishedAt), desc(certificateIssueJobs.requestedAt))
.all()
const byKey = new Map<string, ManagedCertificateTarget>() const byKey = new Map<string, ManagedCertificateTarget>()
for (const row of rows) { for (const row of rows) {
const key = `${row.serverId}::${row.certName}` const key = `${row.serverId}::${row.certName}`
if (byKey.has(key)) continue if (byKey.has(key)) continue
let domainNames: string[] = [] const domainNames = parseJsonArray(row.domainNames).map(String)
try {
const parsed = JSON.parse(row.domainNames) as unknown
if (Array.isArray(parsed)) domainNames = parsed.map(String)
} catch {
domainNames = []
}
byKey.set(key, { byKey.set(key, {
serverId: row.serverId, serverId: row.serverId == null ? "" : String(row.serverId),
certName: row.certName, certName: row.certName,
domainNames, domainNames,
keyType: row.keyType, keyType: row.keyType,
@@ -276,11 +263,10 @@ export function listManagedCertificateTargets(): ManagedCertificateTarget[] {
return [...byKey.values()] return [...byKey.values()]
} }
export function hasActiveIssueJob(serverId: string, certName: string): boolean { export async function hasActiveIssueJob(serverId: string, certName: string): Promise<boolean> {
const row = db.select().from(certificateIssueJobs) const rows = await db.select().from(certificateIssueJobs)
.where(inArray(certificateIssueJobs.status, ["queued", "running"])) .where(inArray(certificateIssueJobs.status, ["queued", "running"]))
.orderBy(desc(certificateIssueJobs.requestedAt)) .orderBy(desc(certificateIssueJobs.requestedAt))
.all() const sid = Number(serverId)
.find((r) => r.serverId === serverId && r.certName === certName) return rows.some((r) => r.serverId === sid && r.certName === certName)
return Boolean(row)
} }
+1 -1
View File
@@ -179,7 +179,7 @@ export async function listFirewallAll(): Promise<{
rules: FirewallRuleDto[] rules: FirewallRuleDto[]
addressLists: FirewallAddressListDto[] addressLists: FirewallAddressListDto[]
}> { }> {
const allServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const allServers = await db.select().from(servers).where(eq(servers.enabled, true))
const perServer = await Promise.all( const perServer = await Promise.all(
allServers.map(async (server) => { allServers.map(async (server) => {
try { try {
@@ -41,8 +41,8 @@ export async function collectGreBgpSnapshotOnce(): Promise<GreBgpSnapshotRunSnap
try { try {
const [greSettled, bgpSettled] = await Promise.allSettled([ const [greSettled, bgpSettled] = await Promise.allSettled([
fetchGreTunnelLiveRows(), await fetchGreTunnelLiveRows(),
fetchBgpSessionsForAlerts(), await fetchBgpSessionsForAlerts(),
]) ])
if (greSettled.status === "rejected") { if (greSettled.status === "rejected") {
+1 -1
View File
@@ -46,7 +46,7 @@ async function fetchGreForServer(server: ServerRow): Promise<GreTunnelLiveRow[]>
/** GRE с включённых серверов — тот же смысл, что `/api/filters/gre-tunnels` (без правил фильтров). */ /** GRE с включённых серверов — тот же смысл, что `/api/filters/gre-tunnels` (без правил фильтров). */
export async function fetchGreTunnelLiveRows(): Promise<GreTunnelLiveRow[]> { export async function fetchGreTunnelLiveRows(): Promise<GreTunnelLiveRow[]> {
const enabledServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const enabledServers = await db.select().from(servers).where(eq(servers.enabled, true))
const results = await Promise.all( const results = await Promise.all(
enabledServers.map(async (server) => { enabledServers.map(async (server) => {
try { try {
+39 -38
View File
@@ -1,4 +1,4 @@
import { and, asc, eq, lt } from "drizzle-orm" import { and, asc, desc, eq, lt } from "drizzle-orm"
import { db } from "../db/index.js" import { db } from "../db/index.js"
import { import {
filterRules, filterRules,
@@ -30,29 +30,29 @@ function norm(v: string | null | undefined): string {
return String(v ?? "").trim().toLowerCase() return String(v ?? "").trim().toLowerCase()
} }
function getSettingsRow() { async function getSettingsRow() {
const row = db.select().from(internetPathSettings).where(eq(internetPathSettings.id, 1)).limit(1).all()[0] const row = (await db.select().from(internetPathSettings).where(eq(internetPathSettings.id, 1)).limit(1))[0]
if (row) return row if (row) return row
const now = new Date().toISOString() const now = new Date().toISOString()
db.insert(internetPathSettings).values({ await db.insert(internetPathSettings).values({
id: 1, id: 1,
enabled: true, enabled: true,
intervalSec: 300, intervalSec: 300,
retentionDays: 14, retentionDays: 14,
createdAt: now, createdAt: now,
updatedAt: now, updatedAt: now,
}).run() })
return db.select().from(internetPathSettings).where(eq(internetPathSettings.id, 1)).limit(1).all()[0] return (await db.select().from(internetPathSettings).where(eq(internetPathSettings.id, 1)).limit(1))[0]
} }
function cleanupSnapshots(retentionDays: number) { async function cleanupSnapshots(retentionDays: number) {
const cutoff = new Date(Date.now() - retentionDays * 24 * 60 * 60 * 1000).toISOString() const cutoff = new Date(Date.now() - retentionDays * 24 * 60 * 60 * 1000).toISOString()
db.delete(internetPathSnapshots).where(lt(internetPathSnapshots.sampledAt, cutoff)).run() await db.delete(internetPathSnapshots).where(lt(internetPathSnapshots.sampledAt, cutoff))
} }
function buildRulesets() { async function buildRulesets() {
const enabled = db.select().from(servers).where(eq(servers.enabled, true)).all() const enabled = await db.select().from(servers).where(eq(servers.enabled, true))
const rules = db.select().from(filterRules).orderBy(asc(filterRules.serverId), asc(filterRules.sortOrder)).all() const rules = await db.select().from(filterRules).orderBy(asc(filterRules.serverId), asc(filterRules.sortOrder))
return enabled.map((s) => ({ return enabled.map((s) => ({
serverId: String(s.id), serverId: String(s.id),
rules: rules rules: rules
@@ -70,7 +70,7 @@ function buildRulesets() {
} }
async function readRouteLookup(serverId: number): Promise<{ gateway: string | null; routingMark: string | null }> { async function readRouteLookup(serverId: number): Promise<{ gateway: string | null; routingMark: string | null }> {
const row = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const row = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!row) return { gateway: null, routingMark: null } if (!row) return { gateway: null, routingMark: null }
const client = MikrotikClient.fromServer(row) const client = MikrotikClient.fromServer(row)
const routes = await client.get<Array<Record<string, string>>>("/ip/route").catch(() => []) const routes = await client.get<Array<Record<string, string>>>("/ip/route").catch(() => [])
@@ -91,8 +91,8 @@ async function readRouteLookup(serverId: number): Promise<{ gateway: string | nu
} }
async function readWanRuntime(serverId: number) { async function readWanRuntime(serverId: number) {
const server = listServersRead().find((s) => Number(s.id) === serverId) const server = (await listServersRead()).find((s) => Number(s.id) === serverId)
const row = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const row = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!server || !row) return null if (!server || !row) return null
const client = MikrotikClient.fromServer(row) const client = MikrotikClient.fromServer(row)
const [dhcpRaw, ipAddrs, routes] = await Promise.all([ const [dhcpRaw, ipAddrs, routes] = await Promise.all([
@@ -144,8 +144,8 @@ async function readWanRuntime(serverId: number) {
} }
} }
function mapSpeedProbes() { async function mapSpeedProbes() {
const rows = db.select().from(uptimeSpeedProbes).orderBy(asc(uptimeSpeedProbes.sortOrder)).all() const rows = await db.select().from(uptimeSpeedProbes).orderBy(asc(uptimeSpeedProbes.sortOrder))
return rows.map((r) => ({ return rows.map((r) => ({
id: r.id, id: r.id,
srcServerId: String(r.srcServerId), srcServerId: String(r.srcServerId),
@@ -175,7 +175,7 @@ function parseInnerIps(comment: string): { localInnerIp: string; remoteInnerIp:
} }
async function collectGreTunnels() { async function collectGreTunnels() {
const enabled = db.select().from(servers).where(eq(servers.enabled, true)).all() const enabled = await db.select().from(servers).where(eq(servers.enabled, true))
const all = await Promise.all(enabled.map(async (srv) => { const all = await Promise.all(enabled.map(async (srv) => {
try { try {
const client = MikrotikClient.fromServer(srv) const client = MikrotikClient.fromServer(srv)
@@ -214,23 +214,24 @@ async function collectGreTunnels() {
return all.flat() return all.flat()
} }
export function getInternetPathSettings() { export async function getInternetPathSettings() {
return getSettingsRow() return await getSettingsRow()
} }
export function updateInternetPathSettings(patch: { enabled?: boolean; intervalSec?: number; retentionDays?: number }) { export async function updateInternetPathSettings(patch: { enabled?: boolean; intervalSec?: number; retentionDays?: number }) {
const prev = getSettingsRow() const prev = await getSettingsRow()
db.update(internetPathSettings).set({ await db.update(internetPathSettings).set({
enabled: patch.enabled ?? prev.enabled, enabled: patch.enabled ?? prev.enabled,
intervalSec: patch.intervalSec ?? prev.intervalSec, intervalSec: patch.intervalSec ?? prev.intervalSec,
retentionDays: patch.retentionDays ?? prev.retentionDays, retentionDays: patch.retentionDays ?? prev.retentionDays,
updatedAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
}).where(eq(internetPathSettings.id, 1)).run() }).where(eq(internetPathSettings.id, 1))
return getSettingsRow() return await getSettingsRow()
} }
export function getLatestInternetPathSnapshot() { export async function getLatestInternetPathSnapshot() {
return db.select().from(internetPathSnapshots).orderBy(asc(internetPathSnapshots.id)).all().at(-1) ?? null const rows = await db.select().from(internetPathSnapshots).orderBy(desc(internetPathSnapshots.id)).limit(1)
return rows[0] ?? null
} }
export async function collectInternetPathSnapshotOnce(): Promise<InternetPathRunSnapshot> { export async function collectInternetPathSnapshotOnce(): Promise<InternetPathRunSnapshot> {
@@ -246,12 +247,12 @@ export async function collectInternetPathSnapshotOnce(): Promise<InternetPathRun
} }
collecting = true collecting = true
const started = Date.now() const started = Date.now()
const settings = getSettingsRow() const settings = await getSettingsRow()
try { try {
const serversRead = listServersRead() const serversRead = await listServersRead()
const homes = serversRead.filter((s) => s.type === "home-router") const homes = serversRead.filter((s) => s.type === "home-router")
const [greTunnels, rulesets] = await Promise.all([collectGreTunnels(), Promise.resolve(buildRulesets())]) const [greTunnels, rulesets] = await Promise.all([await collectGreTunnels(), Promise.resolve(await buildRulesets())])
const speedProbes = mapSpeedProbes() const speedProbes = await mapSpeedProbes()
const routeLookupByServerId: Record<string, { gateway: string | null; routingMark: string | null }> = {} const routeLookupByServerId: Record<string, { gateway: string | null; routingMark: string | null }> = {}
const wanRuntimeByHomeId: Record<string, unknown> = {} const wanRuntimeByHomeId: Record<string, unknown> = {}
for (const h of homes) { for (const h of homes) {
@@ -268,17 +269,17 @@ export async function collectInternetPathSnapshotOnce(): Promise<InternetPathRun
routeLookupByServerId, routeLookupByServerId,
wanRuntimeByHomeId, wanRuntimeByHomeId,
} }
db.insert(internetPathSnapshots).values({ await db.insert(internetPathSnapshots).values({
sampledAt, sampledAt,
payloadJson: JSON.stringify(payload), payloadJson: payload,
}).run() })
cleanupSnapshots(Math.max(1, settings.retentionDays)) await cleanupSnapshots(Math.max(1, settings.retentionDays))
db.update(internetPathSettings).set({ await db.update(internetPathSettings).set({
lastCollectedAt: sampledAt, lastCollectedAt: sampledAt,
lastDurationMs: Date.now() - started, lastDurationMs: Date.now() - started,
lastError: "", lastError: "",
updatedAt: sampledAt, updatedAt: sampledAt,
}).where(eq(internetPathSettings.id, 1)).run() }).where(eq(internetPathSettings.id, 1))
return { return {
v: SCHEDULER_RUN_SNAPSHOT_VERSION, v: SCHEDULER_RUN_SNAPSHOT_VERSION,
job: "internet_path", job: "internet_path",
@@ -288,12 +289,12 @@ export async function collectInternetPathSnapshotOnce(): Promise<InternetPathRun
} }
} catch (e) { } catch (e) {
const msg = e instanceof Error ? e.message : String(e) const msg = e instanceof Error ? e.message : String(e)
db.update(internetPathSettings).set({ await db.update(internetPathSettings).set({
lastCollectedAt: sampledAt, lastCollectedAt: sampledAt,
lastDurationMs: Date.now() - started, lastDurationMs: Date.now() - started,
lastError: msg, lastError: msg,
updatedAt: sampledAt, updatedAt: sampledAt,
}).where(eq(internetPathSettings.id, 1)).run() }).where(eq(internetPathSettings.id, 1))
return { return {
v: SCHEDULER_RUN_SNAPSHOT_VERSION, v: SCHEDULER_RUN_SNAPSHOT_VERSION,
job: "internet_path", job: "internet_path",
+13 -11
View File
@@ -1,5 +1,6 @@
import { eq } from "drizzle-orm" import { eq, lt } from "drizzle-orm"
import { db } from "../db/index.js" import { db, pool } from "../db/index.js"
import { dropExpiredPartitions, ensurePartitionsAround } from "../db/partitions.js"
import { servers, serverSnapshots } from "../db/schema.js" import { servers, serverSnapshots } from "../db/schema.js"
import type { SnapshotInsert } from "../db/schema.js" import type { SnapshotInsert } from "../db/schema.js"
import type { SnapshotRead } from "../types/server.js" import type { SnapshotRead } from "../types/server.js"
@@ -14,12 +15,11 @@ import { invalidateFlowCatalogCache } from "./traffic-flow-topology.js"
* and sync the server's display name from system/identity. * and sync the server's display name from system/identity.
*/ */
export async function pollServer(serverId: number): Promise<SnapshotRead> { export async function pollServer(serverId: number): Promise<SnapshotRead> {
const server = db const server = (await db
.select() .select()
.from(servers) .from(servers)
.where(eq(servers.id, serverId)) .where(eq(servers.id, serverId))
.limit(1) .limit(1))[0]
.all()[0]
if (!server) { if (!server) {
throw new Error(`Server with id=${serverId} not found`) throw new Error(`Server with id=${serverId} not found`)
@@ -61,15 +61,14 @@ export async function pollServer(serverId: number): Promise<SnapshotRead> {
cpuLoad, cpuLoad,
freeMemory: freeMem, freeMemory: freeMem,
totalMemory: totalMem, totalMemory: totalMem,
rawInterfaces: JSON.stringify(ifaces), rawInterfaces: ifaces,
rawIpAddresses: JSON.stringify(addresses), rawIpAddresses: addresses,
} satisfies Partial<SnapshotInsert>) } satisfies Partial<SnapshotInsert>)
if ((identity.name || "") !== (server.name || "")) { if ((identity.name || "") !== (server.name || "")) {
db.update(servers) await db.update(servers)
.set({ name: identity.name, updatedAt: now }) .set({ name: identity.name, updatedAt: now })
.where(eq(servers.id, serverId)) .where(eq(servers.id, serverId))
.run()
invalidateFlowCatalogCache() invalidateFlowCatalogCache()
} }
@@ -78,11 +77,14 @@ export async function pollServer(serverId: number): Promise<SnapshotRead> {
console.warn(`[poller] server id=${serverId} unreachable:`, (err as Error).message) console.warn(`[poller] server id=${serverId} unreachable:`, (err as Error).message)
} }
const [inserted] = db const [inserted] = await db
.insert(serverSnapshots) .insert(serverSnapshots)
.values(partialSnap as SnapshotInsert) .values(partialSnap as SnapshotInsert)
.returning() .returning()
.all()
const cutoffIso = new Date(Date.now() - 14 * 24 * 3600_000).toISOString()
await db.delete(serverSnapshots).where(lt(serverSnapshots.polledAt, cutoffIso))
void dropExpiredPartitions(pool).then(() => ensurePartitionsAround(pool))
return toSnapshotRead(inserted) return toSnapshotRead(inserted)
} }
+37 -40
View File
@@ -6,7 +6,7 @@
* 2. `traffic`, `uptime_resources`, `servers_rest_ping` средний (REST `/system/identity` по каталогу). * 2. `traffic`, `uptime_resources`, `servers_rest_ping` средний (REST `/system/identity` по каталогу).
* 3. `uptime_speed` ниже (BW-test, тяжёлый); локи узлов `withBtestNodeLocks` в speed-сервисе. * 3. `uptime_speed` ниже (BW-test, тяжёлый); локи узлов `withBtestNodeLocks` в speed-сервисе.
* *
* **Оповещения (`alert_engine`):** читают SQLite после джоб сбора (см. `buildSignalSnapshot`), в т.ч. * **Оповещения (`alert_engine`):** читают PostgreSQL после джоб сбора (см. `buildSignalSnapshot`), в т.ч.
* `gre_bgp` snapshot в `scheduler_runs.result_json` (`greTunnels` / `bgpPeers`). После успешного завершения джоб * `gre_bgp` snapshot в `scheduler_runs.result_json` (`greTunnels` / `bgpPeers`). После успешного завершения джоб
* `traffic`, `uptime_*`, `servers_rest_ping`, `gre_bgp` планируется **дополнительный** прогон движка * `traffic`, `uptime_*`, `servers_rest_ping`, `gre_bgp` планируется **дополнительный** прогон движка
* (debounce), см. [`alert-collector-hooks.ts`](./alert-collector-hooks.ts); любой другой писатель сэмплов * (debounce), см. [`alert-collector-hooks.ts`](./alert-collector-hooks.ts); любой другой писатель сэмплов
@@ -89,7 +89,7 @@ function newRunId(): string {
return `sch-${Date.now()}-${Math.random().toString(36).slice(2, 10)}` return `sch-${Date.now()}-${Math.random().toString(36).slice(2, 10)}`
} }
function appendSchedulerRun(row: { async function appendSchedulerRun(row: {
jobKey: string jobKey: string
startedAt: string startedAt: string
finishedAt: string finishedAt: string
@@ -99,8 +99,8 @@ function appendSchedulerRun(row: {
result?: SchedulerRunSnapshot | null result?: SchedulerRunSnapshot | null
}) { }) {
const cutoff = new Date(Date.now() - RUN_LOG_RETENTION_MS).toISOString() const cutoff = new Date(Date.now() - RUN_LOG_RETENTION_MS).toISOString()
db.transaction((tx) => { await db.transaction(async (tx) => {
tx.insert(schedulerRuns).values({ await tx.insert(schedulerRuns).values({
id: newRunId(), id: newRunId(),
jobKey: row.jobKey, jobKey: row.jobKey,
startedAt: row.startedAt, startedAt: row.startedAt,
@@ -108,10 +108,10 @@ function appendSchedulerRun(row: {
status: row.status, status: row.status,
error: row.error, error: row.error,
durationMs: row.durationMs, durationMs: row.durationMs,
resultJson: row.result ? JSON.stringify(row.result) : null, resultJson: row.result ?? null,
}).run() })
tx.delete(schedulerRuns).where(lt(schedulerRuns.finishedAt, cutoff)).run() await tx.delete(schedulerRuns).where(lt(schedulerRuns.finishedAt, cutoff))
tx.delete(events).where(lt(events.createdAt, cutoff)).run() await tx.delete(events).where(lt(events.createdAt, cutoff))
}) })
} }
@@ -167,7 +167,7 @@ async function runSchedulerJobBody(jobKey: SchedulerJobKey): Promise<void> {
throw new Error(`Unknown job: ${jobKey}`) throw new Error(`Unknown job: ${jobKey}`)
} }
const finishedIso = new Date().toISOString() const finishedIso = new Date().toISOString()
appendSchedulerRun({ await appendSchedulerRun({
jobKey, jobKey,
startedAt: startedIso, startedAt: startedIso,
finishedAt: finishedIso, finishedAt: finishedIso,
@@ -177,7 +177,7 @@ async function runSchedulerJobBody(jobKey: SchedulerJobKey): Promise<void> {
result: snapshot ?? null, result: snapshot ?? null,
}) })
if (shouldAppendSchedulerOkEvent(jobKey)) { if (shouldAppendSchedulerOkEvent(jobKey)) {
appendEvent({ await appendEvent({
level: "info", level: "info",
eventType: "scheduler.job.ok", eventType: "scheduler.job.ok",
sourceModule: "scheduler", sourceModule: "scheduler",
@@ -205,7 +205,7 @@ async function runSchedulerJobBody(jobKey: SchedulerJobKey): Promise<void> {
} catch (e) { } catch (e) {
const msg = e instanceof Error ? e.message : String(e) const msg = e instanceof Error ? e.message : String(e)
const finishedIso = new Date().toISOString() const finishedIso = new Date().toISOString()
appendSchedulerRun({ await appendSchedulerRun({
jobKey, jobKey,
startedAt: startedIso, startedAt: startedIso,
finishedAt: finishedIso, finishedAt: finishedIso,
@@ -214,7 +214,7 @@ async function runSchedulerJobBody(jobKey: SchedulerJobKey): Promise<void> {
durationMs: Date.now() - startedAt, durationMs: Date.now() - startedAt,
result: snapshot ?? null, result: snapshot ?? null,
}) })
appendEvent({ await appendEvent({
level: "critical", level: "critical",
eventType: "scheduler.job.failed", eventType: "scheduler.job.failed",
sourceModule: "scheduler", sourceModule: "scheduler",
@@ -267,10 +267,10 @@ function clearAllTimers() {
} }
/** Пересоздать интервалы после смены настроек. */ /** Пересоздать интервалы после смены настроек. */
export function refreshScheduler(): void { export async function refreshScheduler(): Promise<void> {
clearAllTimers() clearAllTimers()
const traffic = getTrafficSettings() const traffic = await getTrafficSettings()
if (traffic.enabled) { if (traffic.enabled) {
const ms = Math.max(5_000, traffic.intervalSec * 1000) const ms = Math.max(5_000, traffic.intervalSec * 1000)
void executeSchedulerJob("traffic").catch(() => {}) void executeSchedulerJob("traffic").catch(() => {})
@@ -282,8 +282,8 @@ export function refreshScheduler(): void {
) )
} }
const apiPing = getServersApiPingSettings() const apiPing = await getServersApiPingSettings()
const internetPath = getInternetPathSettings() const internetPath = await getInternetPathSettings()
if (apiPing.enabled) { if (apiPing.enabled) {
const apiMs = Math.max(10_000, apiPing.intervalSec * 1000) const apiMs = Math.max(10_000, apiPing.intervalSec * 1000)
void executeSchedulerJob("servers_rest_ping").catch(() => {}) void executeSchedulerJob("servers_rest_ping").catch(() => {})
@@ -295,7 +295,7 @@ export function refreshScheduler(): void {
) )
} }
const uptime = getUptimeSettings() const uptime = await getUptimeSettings()
const resOn = uptime.resourcesEnabled ?? uptime.enabled const resOn = uptime.resourcesEnabled ?? uptime.enabled
const pingOn = uptime.pingEnabled ?? uptime.enabled const pingOn = uptime.pingEnabled ?? uptime.enabled
const spdOn = uptime.speedEnabled ?? uptime.enabled const spdOn = uptime.speedEnabled ?? uptime.enabled
@@ -353,7 +353,7 @@ export function refreshScheduler(): void {
}, greBgpMs), }, greBgpMs),
) )
const certRenew = getCertificateRenewSettings() const certRenew = await getCertificateRenewSettings()
if (certRenew.enabled) { if (certRenew.enabled) {
const certRenewMs = Math.max(300_000, certRenew.intervalSec * 1000) const certRenewMs = Math.max(300_000, certRenew.intervalSec * 1000)
void executeSchedulerJob("certificates_renew").catch(() => {}) void executeSchedulerJob("certificates_renew").catch(() => {})
@@ -365,7 +365,7 @@ export function refreshScheduler(): void {
) )
} }
const backupSchedule = getBackupScheduleSettings() const backupSchedule = await getBackupScheduleSettings()
if (backupSchedule.enabled) { if (backupSchedule.enabled) {
const backupMs = 60_000 const backupMs = 60_000
void executeSchedulerJob("backups").catch(() => {}) void executeSchedulerJob("backups").catch(() => {})
@@ -395,21 +395,20 @@ export function isJobRunning(jobKey: string): boolean {
return isSchedulerJobRunning(jobKey) return isSchedulerJobRunning(jobKey)
} }
function lastRunForJob(jobKey: string) { async function lastRunForJob(jobKey: string) {
return db.select().from(schedulerRuns) return (await db.select().from(schedulerRuns)
.where(eq(schedulerRuns.jobKey, jobKey)) .where(eq(schedulerRuns.jobKey, jobKey))
.orderBy(desc(schedulerRuns.finishedAt)) .orderBy(desc(schedulerRuns.finishedAt))
.limit(1) .limit(1))[0]
.all()[0]
} }
export function getSchedulerStatus() { export async function getSchedulerStatus() {
const traffic = getTrafficSettings() const traffic = await getTrafficSettings()
const uptime = getUptimeSettings() const uptime = await getUptimeSettings()
const apiPing = getServersApiPingSettings() const apiPing = await getServersApiPingSettings()
const internetPath = getInternetPathSettings() const internetPath = await getInternetPathSettings()
const certRenew = getCertificateRenewSettings() const certRenew = await getCertificateRenewSettings()
const backupSchedule = getBackupScheduleSettings() const backupSchedule = await getBackupScheduleSettings()
const resOn = uptime.resourcesEnabled ?? uptime.enabled const resOn = uptime.resourcesEnabled ?? uptime.enabled
const pingOn = uptime.pingEnabled ?? uptime.enabled const pingOn = uptime.pingEnabled ?? uptime.enabled
@@ -429,8 +428,8 @@ export function getSchedulerStatus() {
} }
return { return {
jobs: JOB_KEYS.map((jobKey) => { jobs: await Promise.all(JOB_KEYS.map(async (jobKey) => {
const last = lastRunForJob(jobKey) const last = await lastRunForJob(jobKey)
const m = jobMeta[jobKey] const m = jobMeta[jobKey]
return { return {
jobKey, jobKey,
@@ -442,29 +441,27 @@ export function getSchedulerStatus() {
lastDurationMs: last?.durationMs ?? null, lastDurationMs: last?.durationMs ?? null,
lastError: last?.error ?? null, lastError: last?.error ?? null,
} }
}), })),
} }
} }
export function listSchedulerRuns(opts: { jobKey?: string; limit: number; offset: number }) { export async function listSchedulerRuns(opts: { jobKey?: string; limit: number; offset: number }) {
const limit = Math.min(200, Math.max(1, opts.limit)) const limit = Math.min(200, Math.max(1, opts.limit))
const offset = Math.max(0, opts.offset) const offset = Math.max(0, opts.offset)
if (opts.jobKey) { if (opts.jobKey) {
return { return {
runs: db.select().from(schedulerRuns) runs: await db.select().from(schedulerRuns)
.where(eq(schedulerRuns.jobKey, opts.jobKey)) .where(eq(schedulerRuns.jobKey, opts.jobKey))
.orderBy(desc(schedulerRuns.finishedAt)) .orderBy(desc(schedulerRuns.finishedAt))
.limit(limit) .limit(limit)
.offset(offset) .offset(offset),
.all(),
} }
} }
return { return {
runs: db.select().from(schedulerRuns) runs: await db.select().from(schedulerRuns)
.orderBy(desc(schedulerRuns.finishedAt)) .orderBy(desc(schedulerRuns.finishedAt))
.limit(limit) .limit(limit)
.offset(offset) .offset(offset),
.all(),
} }
} }
@@ -1,5 +1,5 @@
import { eq, or } from "drizzle-orm" import { eq, or } from "drizzle-orm"
import { db, sqliteDatabase } from "../db/index.js" import { db, dbAll, dbQuery } from "../db/index.js"
import { servers, serversApiPingSettings, serversRestPingSamples } from "../db/schema.js" import { servers, serversApiPingSettings, serversRestPingSamples } from "../db/schema.js"
import type { ServersRestPingRunSnapshot } from "../types/scheduler-run-snapshot.js" import type { ServersRestPingRunSnapshot } from "../types/scheduler-run-snapshot.js"
import { SCHEDULER_RUN_SNAPSHOT_VERSION } from "../types/scheduler-run-snapshot.js" import { SCHEDULER_RUN_SNAPSHOT_VERSION } from "../types/scheduler-run-snapshot.js"
@@ -7,75 +7,67 @@ import { MikrotikClient } from "./mikrotik.js"
let collecting = false let collecting = false
function getSettingsRow() { async function getSettingsRow() {
const row = db.select().from(serversApiPingSettings).where(eq(serversApiPingSettings.id, 1)).limit(1).all()[0] const row = (await db.select().from(serversApiPingSettings).where(eq(serversApiPingSettings.id, 1)).limit(1))[0]
if (row) return row if (row) return row
const now = new Date().toISOString() const now = new Date().toISOString()
db.insert(serversApiPingSettings).values({ await db.insert(serversApiPingSettings).values({
id: 1, id: 1,
enabled: false, enabled: false,
intervalSec: 120, intervalSec: 120,
createdAt: now, createdAt: now,
updatedAt: now, updatedAt: now,
}).run() })
return db.select().from(serversApiPingSettings).where(eq(serversApiPingSettings.id, 1)).limit(1).all()[0] return (await db.select().from(serversApiPingSettings).where(eq(serversApiPingSettings.id, 1)).limit(1))[0]
} }
/** Как uptime: включённые в каталоге или Home Router (часто «выкл.» при доступе только из LAN). */ /** Как uptime: включённые в каталоге или Home Router (часто «выкл.» при доступе только из LAN). */
function serversForRestProbe() { async function serversForRestProbe() {
return db return await db
.select() .select()
.from(servers) .from(servers)
.where(or(eq(servers.enabled, true), eq(servers.type, "home-router"))) .where(or(eq(servers.enabled, true), eq(servers.type, "home-router")))
.all()
} }
export function getServersApiPingSettings() { export async function getServersApiPingSettings() {
return getSettingsRow() return await getSettingsRow()
} }
export function updateServersApiPingSettings(patch: { enabled?: boolean; intervalSec?: number }) { export async function updateServersApiPingSettings(patch: { enabled?: boolean; intervalSec?: number }) {
const prev = getSettingsRow() const prev = await getSettingsRow()
const next = { const next = {
enabled: patch.enabled ?? prev.enabled, enabled: patch.enabled ?? prev.enabled,
intervalSec: patch.intervalSec ?? prev.intervalSec, intervalSec: patch.intervalSec ?? prev.intervalSec,
updatedAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
} }
db.update(serversApiPingSettings).set(next).where(eq(serversApiPingSettings.id, 1)).run() await db.update(serversApiPingSettings).set(next).where(eq(serversApiPingSettings.id, 1))
return getSettingsRow() return await getSettingsRow()
} }
export function getServersRestPingCollectorState(): { running: boolean } { export function getServersRestPingCollectorState(): { running: boolean } {
return { running: collecting } return { running: collecting }
} }
function loadLastRestSampleByServer(): Map<number, { ok: boolean }> { async function loadLastRestSampleByServer(): Promise<Map<number, { ok: boolean }>> {
const rows = sqliteDatabase const rows = await dbAll<{ serverId: number; ok: boolean | number }>(
.prepare( `SELECT DISTINCT ON (server_id) server_id AS "serverId", ok
`SELECT s.server_id AS serverId, s.ok FROM servers_rest_ping_samples
FROM servers_rest_ping_samples s ORDER BY server_id, sampled_at DESC`,
INNER JOIN ( )
SELECT server_id, MAX(id) AS mid FROM servers_rest_ping_samples GROUP BY server_id
) t ON t.mid = s.id`,
)
.all() as { serverId: number; ok: number }[]
const m = new Map<number, { ok: boolean }>() const m = new Map<number, { ok: boolean }>()
for (const r of rows) m.set(r.serverId, { ok: Boolean(r.ok) }) for (const r of rows) m.set(r.serverId, { ok: Boolean(r.ok) })
return m return m
} }
/** Последний сэмпл ресурсов — для согласования «восстановился» с REST без второго тика uptime. */ /** Последний сэмпл ресурсов — для согласования «восстановился» с REST без второго тика uptime. */
function loadLatestResourceDownServerIds(): Set<number> { async function loadLatestResourceDownServerIds(): Promise<Set<number>> {
const rows = sqliteDatabase const rows = await dbAll<{ serverId: number }>(
.prepare( `SELECT server_id AS "serverId" FROM (
`WITH r AS ( SELECT DISTINCT ON (server_id) server_id, status
SELECT urs.server_id AS serverId, urs.status, FROM uptime_resource_samples
ROW_NUMBER() OVER (PARTITION BY urs.server_id ORDER BY urs.id DESC) AS rn ORDER BY server_id, sampled_at DESC
FROM uptime_resource_samples urs ) latest WHERE status IN ('offline', 'degraded')`,
) )
SELECT serverId FROM r WHERE rn = 1 AND status IN ('offline', 'degraded')`,
)
.all() as { serverId: number }[]
return new Set(rows.map((x) => x.serverId)) return new Set(rows.map((x) => x.serverId))
} }
@@ -104,7 +96,7 @@ export async function collectServersRestPingOnce(): Promise<ServersRestPingRunSn
} }
try { try {
const targets = serversForRestProbe() const targets = await serversForRestProbe()
const rows = await Promise.all( const rows = await Promise.all(
targets.map(async (srv) => { targets.map(async (srv) => {
const name = srv.name || srv.host const name = srv.name || srv.host
@@ -134,23 +126,23 @@ export async function collectServersRestPingOnce(): Promise<ServersRestPingRunSn
) )
snapshot.servers = rows.sort((a, b) => a.serverId - b.serverId) snapshot.servers = rows.sort((a, b) => a.serverId - b.serverId)
const lastByServer = loadLastRestSampleByServer() const lastByServer = await loadLastRestSampleByServer()
const resourceDown = loadLatestResourceDownServerIds() const resourceDown = await loadLatestResourceDownServerIds()
db.transaction((tx) => { await db.transaction(async (tx) => {
for (const row of snapshot.servers) { for (const row of snapshot.servers) {
const sid = row.serverId const sid = row.serverId
const last = lastByServer.get(sid) const last = lastByServer.get(sid)
const errText = row.ok ? null : (row.error ?? null) const errText = row.ok ? null : (row.error ?? null)
const insertOne = (ok: boolean, latencyMs: number | null, error: string | null) => { const insertOne = async (ok: boolean, latencyMs: number | null, error: string | null) => {
tx.insert(serversRestPingSamples).values({ await tx.insert(serversRestPingSamples).values({
serverId: sid, serverId: sid,
sampledAt, sampledAt,
ok, ok,
latencyMs, latencyMs,
error, error,
}).run() })
} }
const lat = row.latencyMs ?? null const lat = row.latencyMs ?? null
@@ -158,23 +150,21 @@ export async function collectServersRestPingOnce(): Promise<ServersRestPingRunSn
if (row.ok) { if (row.ok) {
const downRes = resourceDown.has(sid) const downRes = resourceDown.has(sid)
if (last === undefined && downRes) { if (last === undefined && downRes) {
insertOne(false, null, null) await insertOne(false, null, null)
} }
insertOne(true, lat, null) await insertOne(true, lat, null)
if (last?.ok === false || (last === undefined && downRes)) { if (last?.ok === false || (last === undefined && downRes)) {
insertOne(true, lat, null) await insertOne(true, lat, null)
} }
} else { } else {
insertOne(false, null, errText) await insertOne(false, null, errText)
} }
} }
}) })
sqliteDatabase await dbQuery(`DELETE FROM servers_rest_ping_samples WHERE sampled_at < now() - interval '30 days'`)
.prepare(`DELETE FROM servers_rest_ping_samples WHERE sampled_at < datetime('now', '-30 days')`)
.run()
db.update(serversApiPingSettings) await db.update(serversApiPingSettings)
.set({ .set({
lastCollectedAt: sampledAt, lastCollectedAt: sampledAt,
lastDurationMs: Date.now() - startedAt, lastDurationMs: Date.now() - startedAt,
@@ -182,11 +172,10 @@ export async function collectServersRestPingOnce(): Promise<ServersRestPingRunSn
updatedAt: sampledAt, updatedAt: sampledAt,
}) })
.where(eq(serversApiPingSettings.id, 1)) .where(eq(serversApiPingSettings.id, 1))
.run()
} catch (e) { } catch (e) {
const msg = e instanceof Error ? e.message : String(e) const msg = e instanceof Error ? e.message : String(e)
snapshot.fatalError = msg snapshot.fatalError = msg
db.update(serversApiPingSettings) await db.update(serversApiPingSettings)
.set({ .set({
lastCollectedAt: sampledAt, lastCollectedAt: sampledAt,
lastDurationMs: Date.now() - startedAt, lastDurationMs: Date.now() - startedAt,
@@ -194,7 +183,6 @@ export async function collectServersRestPingOnce(): Promise<ServersRestPingRunSn
updatedAt: sampledAt, updatedAt: sampledAt,
}) })
.where(eq(serversApiPingSettings.id, 1)) .where(eq(serversApiPingSettings.id, 1))
.run()
} finally { } finally {
collecting = false collecting = false
} }
@@ -1,144 +0,0 @@
import assert from "node:assert/strict"
import { randomUUID } from "node:crypto"
import { countSqliteWrites, sqliteDatabase } from "../db/index.js"
import { events } from "../db/schema.js"
import { db } from "../db/index.js"
import {
attachEngineSqlite,
bumpPacketMeta,
configureEngine,
flushPending,
ingestParsedFlowsForServerForTests,
resetEngineForTests,
setEngineError,
} from "./traffic-flow-engine.js"
import { collectGreBgpSnapshotOnce } from "./gre-bgp-snapshot-collector.js"
import { shouldAppendSchedulerOkEvent, executeSchedulerJob } from "./scheduler.js"
import { savePrevLiveMap, loadPrevLiveMap } from "./alert-engine/prev-live-store.js"
import {
invalidateFlowCatalogCache,
loadFlowTopology,
seedFlowTopologyForTests,
} from "./traffic-flow-topology.js"
resetEngineForTests()
attachEngineSqlite(sqliteDatabase)
seedFlowTopologyForTests(null)
invalidateFlowCatalogCache()
const idle1 = countSqliteWrites(() => {
flushPending()
})
assert.equal(idle1.stats.walCheckpoint, 0)
assert.ok(idle1.stats.update >= 1, "first idle flush persists listener stats")
const idle2 = countSqliteWrites(() => {
flushPending()
})
assert.equal(idle2.stats.update, 0, "unchanged listener stats skip UPDATE")
assert.equal(idle2.stats.walCheckpoint, 0)
bumpPacketMeta("203.0.113.9")
const changed = countSqliteWrites(() => {
flushPending()
})
assert.equal(changed.stats.update, 1, "changed packets persist once")
assert.equal(changed.stats.walCheckpoint, 0)
setEngineError("boom")
const errWrite = countSqliteWrites(() => {
flushPending()
})
assert.equal(errWrite.stats.update, 1)
setEngineError("")
flushPending()
resetEngineForTests()
configureEngine({ topN: 20 })
ingestParsedFlowsForServerForTests(9, [{
src: "10.1.1.1",
dst: "8.8.8.8",
proto: 6,
srcPort: 40000,
dstPort: 443,
bytes: 100,
packets: 1,
inIface: "2",
outIface: "",
}])
const withData = countSqliteWrites(() => {
flushPending()
})
assert.equal(withData.stats.walCheckpoint, 0, "flush with data must not TRUNCATE WAL")
assert.ok(withData.stats.insert >= 1, "flow upsert writes")
sqliteDatabase.prepare(`DELETE FROM flow_buckets WHERE server_id = 9`).run()
sqliteDatabase.prepare(`DELETE FROM flow_minute_stats WHERE server_id = 9`).run()
sqliteDatabase.prepare(`DELETE FROM flow_minute_dims WHERE server_id = 9`).run()
sqliteDatabase.prepare(`DELETE FROM flow_daily_dims WHERE server_id = 9`).run()
const plan = sqliteDatabase.prepare(`
EXPLAIN QUERY PLAN
SELECT id FROM flow_buckets WHERE bucket_at >= ? ORDER BY bytes DESC LIMIT 100
`).all("2000-01-01T00:00:00.000Z") as Array<{ detail?: string }>
const planText = plan.map((p) => String(p.detail ?? "")).join(" | ")
assert.ok(planText.length > 0, "EXPLAIN QUERY PLAN returned rows")
invalidateFlowCatalogCache()
seedFlowTopologyForTests(null)
const topoA = loadFlowTopology()
const topoB = loadFlowTopology()
assert.equal(topoA, topoB, "topology cache returns same object")
invalidateFlowCatalogCache()
const topoC = loadFlowTopology()
assert.notEqual(topoA, topoC, "invalidate rebuilds topology")
assert.equal(shouldAppendSchedulerOkEvent("traffic"), false)
assert.equal(shouldAppendSchedulerOkEvent("alert_engine"), false)
assert.equal(shouldAppendSchedulerOkEvent("gre_bgp"), false)
assert.equal(shouldAppendSchedulerOkEvent("backups"), true)
assert.equal(shouldAppendSchedulerOkEvent("certificates_renew"), true)
assert.equal(shouldAppendSchedulerOkEvent("internet_path"), true)
savePrevLiveMap("gre", { a: "up" })
const prevSame = countSqliteWrites(() => {
savePrevLiveMap("gre", { a: "up" })
})
assert.equal(prevSame.stats.update, 0)
assert.equal(prevSame.stats.insert, 0)
savePrevLiveMap("gre", { a: "down" })
assert.equal(loadPrevLiveMap("gre").a, "down")
savePrevLiveMap("gre", { a: "up" })
const greBefore = sqliteDatabase.prepare(`SELECT COUNT(*) AS n FROM alert_gre_tunnel_samples`).get() as { n: number }
const bgpBefore = sqliteDatabase.prepare(`SELECT COUNT(*) AS n FROM alert_bgp_peer_samples`).get() as { n: number }
await collectGreBgpSnapshotOnce()
const greAfter = sqliteDatabase.prepare(`SELECT COUNT(*) AS n FROM alert_gre_tunnel_samples`).get() as { n: number }
const bgpAfter = sqliteDatabase.prepare(`SELECT COUNT(*) AS n FROM alert_bgp_peer_samples`).get() as { n: number }
assert.equal(greAfter.n, greBefore.n)
assert.equal(bgpAfter.n, bgpBefore.n)
const oldId = `evt-old-${randomUUID()}`
db.insert(events).values({
id: oldId,
createdAt: new Date(Date.now() - 40 * 24 * 60 * 60 * 1000).toISOString(),
level: "info",
eventType: "test.retention",
sourceModule: "system",
title: "old",
message: "old",
}).run()
const eventsBefore = sqliteDatabase.prepare(
`SELECT COUNT(*) AS n FROM events WHERE event_type = 'scheduler.job.ok' AND entity_id = 'alert_engine'`,
).get() as { n: number }
await executeSchedulerJob("alert_engine")
const oldGone = sqliteDatabase.prepare(`SELECT COUNT(*) AS n FROM events WHERE id = ?`).get(oldId) as { n: number }
assert.equal(oldGone.n, 0, "events older than 30 days are purged")
const eventsAfter = sqliteDatabase.prepare(
`SELECT COUNT(*) AS n FROM events WHERE event_type = 'scheduler.job.ok' AND entity_id = 'alert_engine'`,
).get() as { n: number }
assert.equal(eventsAfter.n, eventsBefore.n, "quiet jobs do not append scheduler.job.ok")
resetEngineForTests()
console.log("sqlite-write-opt.test.ts: ok")
console.log("EXPLAIN listStoredFlowRows:", planText)
+54 -50
View File
@@ -1,68 +1,78 @@
import { randomUUID } from "node:crypto" import { randomUUID } from "node:crypto"
import { spawn } from "node:child_process"
import { mkdir, readFile, rm, writeFile } from "node:fs/promises" import { mkdir, readFile, rm, writeFile } from "node:fs/promises"
import os from "node:os" import os from "node:os"
import path from "node:path" import path from "node:path"
import Database from "better-sqlite3"
import { env } from "../config.js" import { env } from "../config.js"
import { beginSqliteExclusiveOp, endSqliteExclusiveOp, reopenSqlite, sqliteDatabase } from "../db/index.js" import { withAdvisoryLock } from "../db/index.js"
import { refreshScheduler, stopScheduler } from "./scheduler.js" import { refreshScheduler, stopScheduler } from "./scheduler.js"
import { import {
reattachFlowSqlite,
startTrafficFlowListener, startTrafficFlowListener,
stopTrafficFlowListener, stopTrafficFlowListener,
} from "./traffic-flow-ingest.js" } from "./traffic-flow-ingest.js"
const SQLITE_MAGIC = Buffer.from("SQLite format 3\0") const PG_DUMP_MAGIC = Buffer.from("PGDMP")
const MAX_RESTORE_BYTES = 512 * 1024 * 1024 const MAX_RESTORE_BYTES = 512 * 1024 * 1024
const BACKUP_LOCK_KEY = 8_723_102
type SqliteHandle = InstanceType<typeof Database>
function fmtTimestamp(date = new Date()): string { function fmtTimestamp(date = new Date()): string {
const pad = (n: number) => String(n).padStart(2, "0") const pad = (n: number) => String(n).padStart(2, "0")
return `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}_${pad(date.getHours())}-${pad(date.getMinutes())}-${pad(date.getSeconds())}` return `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}_${pad(date.getHours())}-${pad(date.getMinutes())}-${pad(date.getSeconds())}`
} }
function resolveDatabasePath(): string { function assertPgDumpFile(buffer: Buffer): void {
return path.resolve(process.cwd(), env.DATABASE_PATH) if (buffer.length < PG_DUMP_MAGIC.length) {
} throw new Error("Файл слишком маленький для дампа PostgreSQL")
function assertSqliteFile(buffer: Buffer): void {
if (buffer.length < SQLITE_MAGIC.length) {
throw new Error("Файл слишком маленький для SQLite")
} }
if (!buffer.subarray(0, SQLITE_MAGIC.length).equals(SQLITE_MAGIC)) { if (!buffer.subarray(0, PG_DUMP_MAGIC.length).equals(PG_DUMP_MAGIC)) {
throw new Error("Файл не похож на резервную копию SQLite") throw new Error("Файл не похож на резервную копию PostgreSQL (ожидается формат custom, PGDMP)")
} }
} }
async function withDatabaseOperation<T>(fn: () => Promise<T> | T): Promise<T> { function spawnCollected(command: string, args: string[], opts?: { stdin?: Buffer }): Promise<Buffer> {
beginSqliteExclusiveOp() return new Promise((resolve, reject) => {
stopTrafficFlowListener() const child = spawn(command, args, {
stopScheduler() env: { ...process.env, DATABASE_URL: env.DATABASE_URL },
try { stdio: ["pipe", "pipe", "pipe"],
return await fn() })
} finally { const chunks: Buffer[] = []
startTrafficFlowListener() let err = ""
refreshScheduler() child.stdout.on("data", (chunk: Buffer) => chunks.push(chunk))
endSqliteExclusiveOp() child.stderr.on("data", (chunk: Buffer) => {
} err += chunk.toString("utf8")
})
child.on("error", reject)
child.on("close", (code) => {
if (code !== 0) {
reject(new Error(err.trim() || `${command} завершился с кодом ${code}`))
return
}
resolve(Buffer.concat(chunks))
})
if (opts?.stdin) child.stdin.end(opts.stdin)
else child.stdin.end()
})
}
async function withDatabaseOperation<T>(fn: () => Promise<T>): Promise<T> {
return await withAdvisoryLock(BACKUP_LOCK_KEY, async () => {
stopTrafficFlowListener()
stopScheduler()
try {
return await fn()
} finally {
await startTrafficFlowListener()
await refreshScheduler()
}
})
} }
export async function exportSystemDatabaseBackup(): Promise<{ filename: string; buffer: Buffer }> { export async function exportSystemDatabaseBackup(): Promise<{ filename: string; buffer: Buffer }> {
return withDatabaseOperation(async () => { return await withDatabaseOperation(async () => {
sqliteDatabase.pragma("wal_checkpoint(TRUNCATE)") const buffer = await spawnCollected("pg_dump", ["-Fc", env.DATABASE_URL])
const tempDir = path.join(os.tmpdir(), "mmapp-db-backup") return {
await mkdir(tempDir, { recursive: true }) filename: `mikrotik-manager_${fmtTimestamp()}.dump`,
const tempPath = path.join(tempDir, `manager-${randomUUID()}.db`) buffer,
try {
await sqliteDatabase.backup(tempPath)
const buffer = await readFile(tempPath)
return {
filename: `mikrotik-manager_${fmtTimestamp()}.db`,
buffer,
}
} finally {
await rm(tempPath, { force: true })
} }
}) })
} }
@@ -71,27 +81,21 @@ export async function restoreSystemDatabaseBackup(buffer: Buffer): Promise<void>
if (buffer.length > MAX_RESTORE_BYTES) { if (buffer.length > MAX_RESTORE_BYTES) {
throw new Error("Файл бэкапа слишком большой") throw new Error("Файл бэкапа слишком большой")
} }
assertSqliteFile(buffer) assertPgDumpFile(buffer)
await withDatabaseOperation(async () => { await withDatabaseOperation(async () => {
const tempDir = path.join(os.tmpdir(), "mmapp-db-restore") const tempDir = path.join(os.tmpdir(), "mmapp-db-restore")
await mkdir(tempDir, { recursive: true }) await mkdir(tempDir, { recursive: true })
const tempPath = path.join(tempDir, `restore-${randomUUID()}.db`) const tempPath = path.join(tempDir, `restore-${randomUUID()}.dump`)
let source: SqliteHandle | null = null
try { try {
await writeFile(tempPath, buffer) await writeFile(tempPath, buffer)
source = new Database(tempPath, { readonly: true, fileMustExist: true }) await spawnCollected("pg_restore", ["--clean", "--if-exists", "-d", env.DATABASE_URL, tempPath])
await source.backup(resolveDatabasePath())
reopenSqlite()
reattachFlowSqlite()
sqliteDatabase.pragma("wal_checkpoint(TRUNCATE)")
} finally { } finally {
source?.close()
await rm(tempPath, { force: true }) await rm(tempPath, { force: true })
} }
}) })
} }
export function getSystemDatabasePath(): string { export function getSystemDatabasePath(): string {
return resolveDatabasePath() return env.DATABASE_URL
} }
+31 -35
View File
@@ -64,38 +64,36 @@ function toNum(raw: unknown): number {
return Number.isFinite(n) ? Math.max(0, Math.round(n)) : 0 return Number.isFinite(n) ? Math.max(0, Math.round(n)) : 0
} }
function getSettingsRow() { async function getSettingsRow() {
const row = db.select().from(trafficSettings).where(eq(trafficSettings.id, 1)).limit(1).all()[0] const row = (await db.select().from(trafficSettings).where(eq(trafficSettings.id, 1)).limit(1))[0]
if (row) return row if (row) return row
const now = new Date().toISOString() const now = new Date().toISOString()
db.insert(trafficSettings).values({ await db.insert(trafficSettings).values({
id: 1, id: 1,
enabled: true, enabled: true,
intervalSec: 30, intervalSec: 30,
retentionDays: 14, retentionDays: 14,
createdAt: now, createdAt: now,
updatedAt: now, updatedAt: now,
}).run() })
return db.select().from(trafficSettings).where(eq(trafficSettings.id, 1)).limit(1).all()[0] return (await db.select().from(trafficSettings).where(eq(trafficSettings.id, 1)).limit(1))[0]
} }
function cleanupOldSamples(retentionDays: number) { async function cleanupOldSamples(retentionDays: number) {
const cutoff = new Date(Date.now() - retentionDays * 24 * 60 * 60 * 1000).toISOString() const cutoff = new Date(Date.now() - retentionDays * 24 * 60 * 60 * 1000).toISOString()
db.delete(trafficSamples) await db.delete(trafficSamples)
.where(lt(trafficSamples.sampledAt, cutoff)) .where(lt(trafficSamples.sampledAt, cutoff))
.run()
} }
function readPreviousWave(serverId: number): Map<string, { rxBytes: number; txBytes: number; sampledAt: string }> { async function readPreviousWave(serverId: number): Promise<Map<string, { rxBytes: number; txBytes: number; sampledAt: string }>> {
const last = db const last = (await db
.select({ sampledAt: trafficSamples.sampledAt }) .select({ sampledAt: trafficSamples.sampledAt })
.from(trafficSamples) .from(trafficSamples)
.where(eq(trafficSamples.serverId, serverId)) .where(eq(trafficSamples.serverId, serverId))
.orderBy(desc(trafficSamples.sampledAt)) .orderBy(desc(trafficSamples.sampledAt))
.limit(1) .limit(1))[0]
.all()[0]
if (!last) return new Map() if (!last) return new Map()
const rows = db const rows = await db
.select({ .select({
interfaceName: trafficSamples.interfaceName, interfaceName: trafficSamples.interfaceName,
peerPublicKey: trafficSamples.peerPublicKey, peerPublicKey: trafficSamples.peerPublicKey,
@@ -108,7 +106,6 @@ function readPreviousWave(serverId: number): Map<string, { rxBytes: number; txBy
eq(trafficSamples.serverId, serverId), eq(trafficSamples.serverId, serverId),
eq(trafficSamples.sampledAt, last.sampledAt), eq(trafficSamples.sampledAt, last.sampledAt),
)) ))
.all()
return new Map(rows.map((r) => [`${r.interfaceName}\0${r.peerPublicKey ?? ""}`, r])) return new Map(rows.map((r) => [`${r.interfaceName}\0${r.peerPublicKey ?? ""}`, r]))
} }
@@ -126,7 +123,7 @@ export async function collectTrafficOnce(): Promise<TrafficRunSnapshot> {
collecting = true collecting = true
const startedAt = Date.now() const startedAt = Date.now()
const now = sampledAt const now = sampledAt
const settings = getSettingsRow() const settings = await getSettingsRow()
const snapshot: TrafficRunSnapshot = { const snapshot: TrafficRunSnapshot = {
v: SCHEDULER_RUN_SNAPSHOT_VERSION, v: SCHEDULER_RUN_SNAPSHOT_VERSION,
job: "traffic", job: "traffic",
@@ -135,7 +132,7 @@ export async function collectTrafficOnce(): Promise<TrafficRunSnapshot> {
} }
try { try {
const enabledServers = db.select().from(servers).where(eq(servers.enabled, true)).all() const enabledServers = await db.select().from(servers).where(eq(servers.enabled, true))
for (const srv of enabledServers) { for (const srv of enabledServers) {
const name = srv.name || srv.host const name = srv.name || srv.host
@@ -143,7 +140,7 @@ export async function collectTrafficOnce(): Promise<TrafficRunSnapshot> {
const client = MikrotikClient.fromServer(srv) const client = MikrotikClient.fromServer(srv)
const ifaces = await client.get<RosIfaceTraffic[]>("/interface") const ifaces = await client.get<RosIfaceTraffic[]>("/interface")
rememberServerIfaces(srv.id, ifaces) rememberServerIfaces(srv.id, ifaces)
const prevWave = readPreviousWave(srv.id) const prevWave = await readPreviousWave(srv.id)
const nowMs = Date.parse(now) const nowMs = Date.parse(now)
let sumRxMbps = 0 let sumRxMbps = 0
let sumTxMbps = 0 let sumTxMbps = 0
@@ -205,7 +202,7 @@ export async function collectTrafficOnce(): Promise<TrafficRunSnapshot> {
/* WG peers optional — iface samples already recorded */ /* WG peers optional — iface samples already recorded */
} }
if (rows.length > 0) { if (rows.length > 0) {
db.insert(trafficSamples).values(rows).run() await db.insert(trafficSamples).values(rows)
} }
snapshot.servers.push({ snapshot.servers.push({
serverId: srv.id, serverId: srv.id,
@@ -227,22 +224,22 @@ export async function collectTrafficOnce(): Promise<TrafficRunSnapshot> {
} }
} }
cleanupOldSamples(Math.max(1, settings.retentionDays)) await cleanupOldSamples(Math.max(1, settings.retentionDays))
db.update(trafficSettings).set({ await db.update(trafficSettings).set({
lastCollectedAt: now, lastCollectedAt: now,
lastDurationMs: Date.now() - startedAt, lastDurationMs: Date.now() - startedAt,
lastError: "", lastError: "",
updatedAt: now, updatedAt: now,
}).where(eq(trafficSettings.id, 1)).run() }).where(eq(trafficSettings.id, 1))
} catch (e) { } catch (e) {
const msg = e instanceof Error ? e.message : String(e) const msg = e instanceof Error ? e.message : String(e)
snapshot.fatalError = msg snapshot.fatalError = msg
db.update(trafficSettings).set({ await db.update(trafficSettings).set({
lastCollectedAt: now, lastCollectedAt: now,
lastDurationMs: Date.now() - startedAt, lastDurationMs: Date.now() - startedAt,
lastError: msg, lastError: msg,
updatedAt: now, updatedAt: now,
}).where(eq(trafficSettings.id, 1)).run() }).where(eq(trafficSettings.id, 1))
} finally { } finally {
collecting = false collecting = false
} }
@@ -254,8 +251,8 @@ export function stopTrafficCollector() {}
export function restartTrafficCollector() {} export function restartTrafficCollector() {}
export function getTrafficCollectorState(): TrafficCollectorState { export async function getTrafficCollectorState(): Promise<TrafficCollectorState> {
const s = getSettingsRow() const s = await getSettingsRow()
return { return {
running: Boolean(s.enabled), running: Boolean(s.enabled),
lastRunAt: s.lastCollectedAt ?? null, lastRunAt: s.lastCollectedAt ?? null,
@@ -263,38 +260,37 @@ export function getTrafficCollectorState(): TrafficCollectorState {
} }
} }
/** Идёт активный прогон `collectTrafficOnce` (запись в SQLite ещё не завершена). */ /** Идёт активный прогон `collectTrafficOnce` (запись в PostgreSQL ещё не завершена). */
export function isTrafficCollecting(): boolean { export function isTrafficCollecting(): boolean {
return collecting return collecting
} }
export function getTrafficSettings() { export async function getTrafficSettings() {
return getSettingsRow() return await getSettingsRow()
} }
export function updateTrafficSettings(patch: { export async function updateTrafficSettings(patch: {
enabled?: boolean enabled?: boolean
intervalSec?: number intervalSec?: number
retentionDays?: number retentionDays?: number
}) { }) {
const prev = getSettingsRow() const prev = await getSettingsRow()
const next = { const next = {
enabled: patch.enabled ?? prev.enabled, enabled: patch.enabled ?? prev.enabled,
intervalSec: patch.intervalSec ?? prev.intervalSec, intervalSec: patch.intervalSec ?? prev.intervalSec,
retentionDays: patch.retentionDays ?? prev.retentionDays, retentionDays: patch.retentionDays ?? prev.retentionDays,
updatedAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
} }
db.update(trafficSettings).set(next).where(eq(trafficSettings.id, 1)).run() await db.update(trafficSettings).set(next).where(eq(trafficSettings.id, 1))
return getSettingsRow() return await getSettingsRow()
} }
export function readServerSamplesInRange(serverId: number, sinceIso: string) { export async function readServerSamplesInRange(serverId: number, sinceIso: string) {
return db.select() return await db.select()
.from(trafficSamples) .from(trafficSamples)
.where(and( .where(and(
eq(trafficSamples.serverId, serverId), eq(trafficSamples.serverId, serverId),
gte(trafficSamples.sampledAt, sinceIso), gte(trafficSamples.sampledAt, sinceIso),
)) ))
.orderBy(asc(trafficSamples.sampledAt)) .orderBy(asc(trafficSamples.sampledAt))
.all()
} }
@@ -5,7 +5,8 @@ import {
resetFlowRingsForTests, resetFlowRingsForTests,
} from "./traffic-flow-ingest.js" } from "./traffic-flow-ingest.js"
import { buildFlowAnalytics, formatLiveSseFromBuilder, getFlowMonthly, listFlowClients, listFlowExporters } from "./traffic-flow-analytics.js" import { buildFlowAnalytics, formatLiveSseFromBuilder, getFlowMonthly, listFlowClients, listFlowExporters } from "./traffic-flow-analytics.js"
import { sqliteDatabase } from "../db/index.js" import { withPgOrSkip } from "../test/pg.js"
import { dbQuery } from "../db/index.js"
import { seedFlowTopologyForTests, type FlowTopology } from "./traffic-flow-topology.js" import { seedFlowTopologyForTests, type FlowTopology } from "./traffic-flow-topology.js"
import { disableCatalogFetchForTests, resetFlowCatalogForTests, seedFlowCatalogForTests } from "./traffic-flow-classify.js" import { disableCatalogFetchForTests, resetFlowCatalogForTests, seedFlowCatalogForTests } from "./traffic-flow-classify.js"
import { import {
@@ -15,6 +16,11 @@ import {
seedRipeCacheForTests, seedRipeCacheForTests,
} from "./traffic-flow-ripe.js" } from "./traffic-flow-ripe.js"
if (!(await withPgOrSkip())) {
console.log("traffic-flow-analytics.test.ts: skip")
process.exit(0)
}
disableCatalogFetchForTests() disableCatalogFetchForTests()
resetFlowCatalogForTests() resetFlowCatalogForTests()
disableRipePersistForTests() disableRipePersistForTests()
@@ -63,7 +69,7 @@ ingestParsedFlowsForServerForTests(7, [
]) ])
try { try {
const all = buildFlowAnalytics({ minutes: 5, serverId: 7 }) const all = await buildFlowAnalytics({ minutes: 5, serverId: 7 })
assert.equal(all.applications[0]?.label, "HTTPS") assert.equal(all.applications[0]?.label, "HTTPS")
assert.ok(all.protocols.some((p) => p.label === "TCP")) assert.ok(all.protocols.some((p) => p.label === "TCP"))
assert.equal(all.ifaces[0]?.name, "ether1") assert.equal(all.ifaces[0]?.name, "ether1")
@@ -75,14 +81,14 @@ try {
assert.equal(conv.application, "HTTPS") assert.equal(conv.application, "HTTPS")
assert.ok(!/^\d+$/.test(conv.inIface)) assert.ok(!/^\d+$/.test(conv.inIface))
const filtered = buildFlowAnalytics({ minutes: 5, serverId: 7, iface: "ether1" }) const filtered = await buildFlowAnalytics({ minutes: 5, serverId: 7, iface: "ether1" })
assert.ok(filtered.bytes >= 12_000) assert.ok(filtered.bytes >= 12_000)
assert.equal(filtered.ifaces[0]?.name, "ether1") assert.equal(filtered.ifaces[0]?.name, "ether1")
const miss = buildFlowAnalytics({ minutes: 5, serverId: 7, iface: "wg-flow" }) const miss = await buildFlowAnalytics({ minutes: 5, serverId: 7, iface: "wg-flow" })
assert.equal(miss.conversations, 0) assert.equal(miss.conversations, 0)
const other = buildFlowAnalytics({ minutes: 5, serverId: 99 }) const other = await buildFlowAnalytics({ minutes: 5, serverId: 99 })
assert.equal(other.conversations, 0) assert.equal(other.conversations, 0)
} finally { } finally {
resetFlowRingsForTests() resetFlowRingsForTests()
@@ -121,10 +127,10 @@ ingestParsedFlowsForServerForTests(7, [
}, },
]) ])
try { try {
const summed = buildFlowAnalytics({ minutes: 5, serverId: 7, dedup: false }) const summed = await buildFlowAnalytics({ minutes: 5, serverId: 7, dedup: false })
assert.equal(summed.bytes, 21_000) assert.equal(summed.bytes, 21_000)
assert.equal(summed.conversations, 2) assert.equal(summed.conversations, 2)
const deduped = buildFlowAnalytics({ minutes: 5, serverId: 7, dedup: true }) const deduped = await buildFlowAnalytics({ minutes: 5, serverId: 7, dedup: true })
assert.equal(deduped.bytes, 12_000) assert.equal(deduped.bytes, 12_000)
assert.equal(deduped.conversations, 1) assert.equal(deduped.conversations, 1)
assert.equal(deduped.dedupApplied, true) assert.equal(deduped.dedupApplied, true)
@@ -165,7 +171,7 @@ ingestParsedFlowsForServerForTests(7, [
}, },
]) ])
try { try {
const geo = buildFlowAnalytics({ minutes: 5, serverId: 7 }) const geo = await buildFlowAnalytics({ minutes: 5, serverId: 7 })
assert.equal(geo.categories?.[0]?.label, "Игры") assert.equal(geo.categories?.[0]?.label, "Игры")
assert.ok(geo.asns?.some((r) => r.label.includes("AS15169"))) assert.ok(geo.asns?.some((r) => r.label.includes("AS15169")))
assert.equal(geo.countries?.[0]?.id, "US") assert.equal(geo.countries?.[0]?.id, "US")
@@ -210,7 +216,7 @@ ingestParsedFlowsForServerForTests(7, [
}, },
]) ])
try { try {
const cf = buildFlowAnalytics({ minutes: 5, serverId: 7 }) const cf = await buildFlowAnalytics({ minutes: 5, serverId: 7 })
assert.equal(cf.countries?.[0]?.id, "US") assert.equal(cf.countries?.[0]?.id, "US")
assert.ok(cf.mapEdges?.every((e) => e.toCountry !== "?")) assert.ok(cf.mapEdges?.every((e) => e.toCountry !== "?"))
assert.equal(cf.services?.[0]?.label, "Cloudflare") assert.equal(cf.services?.[0]?.label, "Cloudflare")
@@ -238,7 +244,7 @@ try {
outIface: "", outIface: "",
}, },
]) ])
const degraded = buildFlowAnalytics({ minutes: 5, serverId: 7, skipHeavy: true }) const degraded = await buildFlowAnalytics({ minutes: 5, serverId: 7, skipHeavy: true })
assert.equal(degraded.degraded, true) assert.equal(degraded.degraded, true)
assert.equal(degraded.conversationsList.length, 0) assert.equal(degraded.conversationsList.length, 0)
assert.ok((degraded.bytes ?? 0) >= 12_000) assert.ok((degraded.bytes ?? 0) >= 12_000)
@@ -249,30 +255,35 @@ try {
assert.equal((liveErr.data as { error: string }).error, "SQLITE_BUSY") assert.equal((liveErr.data as { error: string }).error, "SQLITE_BUSY")
const liveOk = formatLiveSseFromBuilder(() => ({ ok: true })) const liveOk = formatLiveSseFromBuilder(() => ({ ok: true }))
assert.equal(liveOk.event, "sample") assert.equal(liveOk.event, "sample")
const exporters = listFlowExporters(5) const exporters = await listFlowExporters(5)
const clients = listFlowClients(5) const clients = await listFlowClients(5)
assert.ok(Array.isArray(exporters.exporters)) assert.ok(Array.isArray(exporters.exporters))
assert.ok(Array.isArray(clients.clients)) assert.ok(Array.isArray(clients.clients))
resetFlowRingsForTests() resetFlowRingsForTests()
} }
{ {
sqliteDatabase.prepare(`DELETE FROM flow_daily_dims WHERE server_id = 7 AND day LIKE '2026-09-%'`).run() if (await withPgOrSkip()) {
sqliteDatabase.exec(` const { pool } = await import("../db/index.js")
INSERT INTO flow_daily_dims (server_id, day, dim, key, bytes, packets) const { ensurePartitionFor } = await import("../db/partitions.js")
VALUES await ensurePartitionFor(pool, "flow_daily_dims", "month", new Date("2026-09-01T00:00:00Z"))
(7, '2026-09-01', 'country', 'US', 1000, 10), await dbQuery(`DELETE FROM flow_daily_dims WHERE server_id = 7 AND day >= '2026-09-01' AND day < '2026-10-01'`)
(7, '2026-09-02', 'country', 'US', 500, 5), await dbQuery(`
(7, '2026-09-01', 'service', 'steam', 800, 8), INSERT INTO flow_daily_dims (server_id, day, dim, key, bytes, packets)
(7, '2026-09-01', 'asn', '15169', 900, 9), VALUES
(7, '2026-09-01', 'asn', 'other', 100, 1) (7, '2026-09-01', 'country', 'US', 1000, 10),
`) (7, '2026-09-02', 'country', 'US', 500, 5),
const monthly = getFlowMonthly("2026-09", 7) (7, '2026-09-01', 'service', 'steam', 800, 8),
assert.equal(monthly.bytes, 1500) (7, '2026-09-01', 'asn', '15169', 900, 9),
assert.equal(monthly.countries[0]?.id, "US") (7, '2026-09-01', 'asn', 'other', 100, 1)
assert.equal(monthly.countries[0]?.bytes, 1500) `)
assert.ok(monthly.asns.some((row) => row.id === "other")) const monthly = await getFlowMonthly("2026-09", 7)
sqliteDatabase.prepare(`DELETE FROM flow_daily_dims WHERE server_id = 7 AND day LIKE '2026-09-%'`).run() assert.equal(monthly.bytes, 1500)
assert.equal(monthly.countries[0]?.id, "US")
assert.equal(monthly.countries[0]?.bytes, 1500)
assert.ok(monthly.asns.some((row) => row.id === "other"))
await dbQuery(`DELETE FROM flow_daily_dims WHERE server_id = 7 AND day >= '2026-09-01' AND day < '2026-10-01'`)
}
} }
{ {
@@ -353,7 +364,7 @@ try {
}, },
]) ])
try { try {
const def = buildFlowAnalytics({ minutes: 5, serverId: 7 }) const def = await buildFlowAnalytics({ minutes: 5, serverId: 7 })
assert.equal(def.bytes, 12_000) assert.equal(def.bytes, 12_000)
assert.equal(def.bytesPayload, 12_000) assert.equal(def.bytesPayload, 12_000)
assert.equal(def.bytesOverlay, 5_000_000) assert.equal(def.bytesOverlay, 5_000_000)
@@ -371,7 +382,7 @@ try {
assert.equal(path.clientName, "Alice") assert.equal(path.clientName, "Alice")
assert.equal(path.enName, "NSK-SERVHOST-RTK") assert.equal(path.enName, "NSK-SERVHOST-RTK")
assert.equal(path.dst, "173.194.160.163") assert.equal(path.dst, "173.194.160.163")
const withAll = buildFlowAnalytics({ minutes: 5, serverId: 7, excludeOverlay: false, excludeMesh: false }) const withAll = await buildFlowAnalytics({ minutes: 5, serverId: 7, excludeOverlay: false, excludeMesh: false })
assert.equal(withAll.bytes, 12_000 + 5_000_000 + 8000) assert.equal(withAll.bytes, 12_000 + 5_000_000 + 8000)
assert.ok(withAll.conversationsList.some((r) => r.plane === "overlay")) assert.ok(withAll.conversationsList.some((r) => r.plane === "overlay"))
assert.ok(withAll.conversationsList.some((r) => r.plane === "client_mesh")) assert.ok(withAll.conversationsList.some((r) => r.plane === "client_mesh"))
@@ -415,7 +426,7 @@ try {
}, },
]) ])
try { try {
const rev = buildFlowAnalytics({ minutes: 5, serverId: 7 }) const rev = await buildFlowAnalytics({ minutes: 5, serverId: 7 })
const google = rev.conversationsList.find((r) => r.src === "173.194.151.65") const google = rev.conversationsList.find((r) => r.src === "173.194.151.65")
const cf = rev.conversationsList.find((r) => r.src === "104.18.35.51") const cf = rev.conversationsList.find((r) => r.src === "104.18.35.51")
assert.equal(google?.service, "Google") assert.equal(google?.service, "Google")
@@ -430,37 +441,42 @@ try {
} }
{ {
resetFlowRingsForTests() if (await withPgOrSkip()) {
resetIfaceCacheForTests() resetFlowRingsForTests()
const sidRow = sqliteDatabase.prepare(`SELECT id FROM servers LIMIT 1`).get() as { id?: number } | undefined resetIfaceCacheForTests()
if (sidRow?.id) { const sidRow = await dbQuery<{ id: number }>(`SELECT id FROM servers LIMIT 1`)
const sid = sidRow.id const sid = sidRow.rows[0]?.id
rememberServerIfaces(sid, [{ ".id": "*4", name: "gre-en-nsk" }]) if (sid) {
ingestParsedFlowsForServerForTests(sid, [{ const { pool } = await import("../db/index.js")
src: "10.100.1.17", const { ensurePartitionFor } = await import("../db/partitions.js")
dst: "8.8.8.8", await ensurePartitionFor(pool, "traffic_samples", "week", new Date())
proto: 6, rememberServerIfaces(sid, [{ ".id": "*4", name: "gre-en-nsk" }])
srcPort: 1, ingestParsedFlowsForServerForTests(sid, [{
dstPort: 443, src: "10.100.1.17",
bytes: 100, dst: "8.8.8.8",
packets: 1, proto: 6,
inIface: "4", srcPort: 1,
outIface: "4", dstPort: 443,
}]) bytes: 100,
sqliteDatabase.prepare(` packets: 1,
INSERT INTO traffic_samples (server_id, interface_name, sampled_at, rx_bytes, tx_bytes, rx_bps, tx_bps) inIface: "4",
VALUES (?, 'gre-en-nsk', datetime('now'), 9000000, 1000000, 40000000, 2000000) outIface: "4",
`).run(sid) }])
try { await dbQuery(`
const wire = buildFlowAnalytics({ minutes: 5, serverId: sid }) INSERT INTO traffic_samples (server_id, interface_name, sampled_at, rx_bytes, tx_bytes, rx_bps, tx_bps)
assert.ok((wire.bpsWire ?? 0) >= 40_000_000) VALUES ($1, 'gre-en-nsk', now(), 9000000, 1000000, 40000000, 2000000)
assert.notEqual(wire.bpsWire, (wire.bytes * 8) / 300) `, [sid])
} finally { try {
sqliteDatabase.prepare(`DELETE FROM traffic_samples WHERE server_id = ? AND interface_name = 'gre-en-nsk'`).run(sid) const wire = await buildFlowAnalytics({ minutes: 5, serverId: sid })
assert.ok((wire.bpsWire ?? 0) >= 40_000_000)
assert.notEqual(wire.bpsWire, (wire.bytes * 8) / 300)
} finally {
await dbQuery(`DELETE FROM traffic_samples WHERE server_id = $1 AND interface_name = 'gre-en-nsk'`, [sid])
}
} }
resetFlowRingsForTests()
resetIfaceCacheForTests()
} }
resetFlowRingsForTests()
resetIfaceCacheForTests()
} }
console.log("traffic-flow-analytics.test.ts: ok") console.log("traffic-flow-analytics.test.ts: ok")
+26 -26
View File
@@ -1,5 +1,5 @@
import { eq } from "drizzle-orm" import { eq } from "drizzle-orm"
import { db, sqliteDatabase } from "../db/index.js" import { db, dbAll } from "../db/index.js"
import { appUsers, userInterfaceBindings } from "../db/schema.js" import { appUsers, userInterfaceBindings } from "../db/schema.js"
import type { import type {
FlowAnalyticsDto, FlowAnalyticsDto,
@@ -94,9 +94,9 @@ function bump(
map.set(id, prev) map.set(id, prev)
} }
function userIfaceAllow(userId: string): Map<number, Set<string>> | null { async function userIfaceAllow(userId: string): Promise<Map<number, Set<string>> | null> {
if (!userId) return null if (!userId) return null
const binds = db.select().from(userInterfaceBindings).where(eq(userInterfaceBindings.userId, userId)).all() const binds = await db.select().from(userInterfaceBindings).where(eq(userInterfaceBindings.userId, userId))
const allow = new Map<number, Set<string>>() const allow = new Map<number, Set<string>>()
for (const b of binds) { for (const b of binds) {
const set = allow.get(b.serverId) ?? new Set<string>() const set = allow.get(b.serverId) ?? new Set<string>()
@@ -142,13 +142,13 @@ function topLabel(map: Map<string, { bytes: number; packets: number; label?: str
return best return best
} }
export function buildFlowAnalytics(q: FlowAnalyticsQuery): FlowAnalyticsDto { export async function buildFlowAnalytics(q: FlowAnalyticsQuery): Promise<FlowAnalyticsDto> {
const key = analyticsQueryKey(q) const key = analyticsQueryKey(q)
const now = Date.now() const now = Date.now()
if (analyticsCache && analyticsCache.key === key && now - analyticsCache.at < ANALYTICS_CACHE_TTL_MS) { if (analyticsCache && analyticsCache.key === key && now - analyticsCache.at < ANALYTICS_CACHE_TTL_MS) {
return analyticsCache.dto return analyticsCache.dto
} }
const dto = buildFlowAnalyticsUncached(q) const dto = await buildFlowAnalyticsUncached(q)
analyticsCache = { key, at: now, dto } analyticsCache = { key, at: now, dto }
return dto return dto
} }
@@ -174,20 +174,20 @@ function analyticsQueryKey(q: FlowAnalyticsQuery): string {
const ANALYTICS_CACHE_TTL_MS = 2000 const ANALYTICS_CACHE_TTL_MS = 2000
let analyticsCache: { key: string; at: number; dto: FlowAnalyticsDto } | null = null let analyticsCache: { key: string; at: number; dto: FlowAnalyticsDto } | null = null
function buildFlowAnalyticsUncached(q: FlowAnalyticsQuery): FlowAnalyticsDto { async function buildFlowAnalyticsUncached(q: FlowAnalyticsQuery): Promise<FlowAnalyticsDto> {
const settings = getTrafficFlowSettingsRow() const settings = await getTrafficFlowSettingsRow()
const top = Math.min(50, Math.max(10, settings.topN)) const top = Math.min(50, Math.max(10, settings.topN))
const windowSec = Math.max(60, q.minutes * 60) const windowSec = Math.max(60, q.minutes * 60)
const raw = listFlowRowsForWindow(q.minutes) const raw = await listFlowRowsForWindow(q.minutes)
const allow = q.userId ? userIfaceAllow(q.userId) : null const allow = q.userId ? await userIfaceAllow(q.userId) : null
const catalog = getServerCatalog() const catalog = await getServerCatalog()
const nameById = new Map([...catalog.byId].map(([id, s]) => [id, s.name])) const nameById = new Map([...catalog.byId].map(([id, s]) => [id, s.name]))
const countryById = new Map([...catalog.byId].map(([id, s]) => [id, s.country])) const countryById = new Map([...catalog.byId].map(([id, s]) => [id, s.country]))
const ifaceFilter = q.iface && q.iface !== "__all__" ? q.iface : "" const ifaceFilter = q.iface && q.iface !== "__all__" ? q.iface : ""
const wantDedup = q.dedup !== false && !ifaceFilter const wantDedup = q.dedup !== false && !ifaceFilter
const excludeMesh = q.excludeMesh !== false const excludeMesh = q.excludeMesh !== false
const excludeOverlay = q.excludeOverlay !== false const excludeOverlay = q.excludeOverlay !== false
const topo = loadFlowTopology() const topo = await loadFlowTopology()
refreshFlowCatalogInBackground() refreshFlowCatalogInBackground()
@@ -447,7 +447,7 @@ function buildFlowAnalyticsUncached(q: FlowAnalyticsQuery): FlowAnalyticsDto {
? getRingMbps(ringServer, RING_OVERLAY) ? getRingMbps(ringServer, RING_OVERLAY)
: { rxNow: 0, txNow: 0 } : { rxNow: 0, txNow: 0 }
const greNames = ringServer ? enGreIfaceNames(topo, ringServer, [...ifacesForWire]) : [] const greNames = ringServer ? enGreIfaceNames(topo, ringServer, [...ifacesForWire]) : []
const wire = ringServer ? latestWireBps(ringServer, greNames) : { bps: 0, bytes: 0 } const wire = ringServer ? await latestWireBps(ringServer, greNames) : { bps: 0, bytes: 0 }
return { return {
bpsNow: (ring.rxNow + ring.txNow) * 1_000_000 || (totalBytes * 8) / windowSec, bpsNow: (ring.rxNow + ring.txNow) * 1_000_000 || (totalBytes * 8) / windowSec,
@@ -505,13 +505,13 @@ function summarizeByServer(rows: PendingFlowRow[]) {
return { bytes, sessions } return { bytes, sessions }
} }
export function listFlowExporters(minutes: number): FlowExportersDto { export async function listFlowExporters(minutes: number): Promise<FlowExportersDto> {
const runtime = getFlowRuntimeCounters() const runtime = await getFlowRuntimeCounters()
const rows = listFlowRowsForWindow(minutes) const rows = await listFlowRowsForWindow(minutes)
const { bytes, sessions } = summarizeByServer(rows) const { bytes, sessions } = summarizeByServer(rows)
const ids = new Set<number>([...bytes.keys()]) const ids = new Set<number>([...bytes.keys()])
for (const p of listHostPeers()) ids.add(p.serverId) for (const p of await listHostPeers()) ids.add(p.serverId)
const catalog = getServerCatalog() const catalog = await getServerCatalog()
const emptySeries = Array(60).fill(0) as number[] const emptySeries = Array(60).fill(0) as number[]
const exporters = catalog.list const exporters = catalog.list
.filter((s) => ids.has(s.id)) .filter((s) => ids.has(s.id))
@@ -546,16 +546,16 @@ export function listFlowExporters(minutes: number): FlowExportersDto {
} }
} }
export function listFlowClients(minutes: number): FlowClientsDto { export async function listFlowClients(minutes: number): Promise<{ clients: { id: string; name: string; subtitle: string; site: string; country: string; status: "online" | "offline" | "degraded"; rxNow: number; txNow: number; sessions: number; rxSeries: number[]; txSeries: number[]; bytes: number; }[]; }> {
const users = db.select().from(appUsers).all() const users = await db.select().from(appUsers)
const binds = db.select().from(userInterfaceBindings).all() const binds = await db.select().from(userInterfaceBindings)
const byUser = new Map<string, typeof binds>() const byUser = new Map<string, typeof binds>()
for (const b of binds) { for (const b of binds) {
const list = byUser.get(b.userId) ?? [] const list = byUser.get(b.userId) ?? []
list.push(b) list.push(b)
byUser.set(b.userId, list) byUser.set(b.userId, list)
} }
const rows = listFlowRowsForWindow(minutes) const rows = await listFlowRowsForWindow(minutes)
const emptySeries = Array(60).fill(0) as number[] const emptySeries = Array(60).fill(0) as number[]
const windowSec = Math.max(60, minutes * 60) const windowSec = Math.max(60, minutes * 60)
const clients: FlowEntityCard[] = [] const clients: FlowEntityCard[] = []
@@ -617,9 +617,9 @@ export function safeBuildLiveFlowSample(q: Omit<FlowAnalyticsQuery, "minutes" |
event: "sample" | "error" event: "sample" | "error"
data: unknown data: unknown
} { } {
return formatLiveSseFromBuilder(() => { return formatLiveSseFromBuilder(async () => {
const skipHeavy = isFlowAnalyticsDegraded() const skipHeavy = isFlowAnalyticsDegraded()
return buildFlowAnalytics({ ...q, minutes: LIVE_ANALYTICS_MINUTES, skipHeavy }) return await buildFlowAnalytics({ ...q, minutes: LIVE_ANALYTICS_MINUTES, skipHeavy })
}) })
} }
@@ -653,7 +653,7 @@ function toBreakdown(
})) }))
} }
export function getFlowMonthly(month: string, serverId?: number): FlowMonthlyDto { export async function getFlowMonthly(month: string, serverId?: number): Promise<FlowMonthlyDto> {
const bounds = monthBounds(month) const bounds = monthBounds(month)
if (!bounds) { if (!bounds) {
return { month, bytes: 0, countries: [], services: [], asns: [] } return { month, bytes: 0, countries: [], services: [], asns: [] }
@@ -664,12 +664,12 @@ export function getFlowMonthly(month: string, serverId?: number): FlowMonthlyDto
where += " AND server_id = ?" where += " AND server_id = ?"
params.push(serverId) params.push(serverId)
} }
const rows = sqliteDatabase.prepare(` const rows = await dbAll<{ dim: string; key: string; bytes: number; packets: number }>(`
SELECT dim AS dim, key AS key, SUM(bytes) AS bytes, SUM(packets) AS packets SELECT dim AS dim, key AS key, SUM(bytes) AS bytes, SUM(packets) AS packets
FROM flow_daily_dims FROM flow_daily_dims
WHERE ${where} WHERE ${where}
GROUP BY dim, key GROUP BY dim, key
`).all(...params) as Array<{ dim: string; key: string; bytes: number; packets: number }> `, params)
const countries: Array<{ key: string; bytes: number; packets: number }> = [] const countries: Array<{ key: string; bytes: number; packets: number }> = []
const services: Array<{ key: string; bytes: number; packets: number }> = [] const services: Array<{ key: string; bytes: number; packets: number }> = []
@@ -97,7 +97,7 @@ async function fetchCatalog(): Promise<void> {
if (inflight) return inflight if (inflight) return inflight
inflight = (async () => { inflight = (async () => {
try { try {
const row = db.select().from(evobgpSettings).limit(1).all()[0] const row = (await db.select().from(evobgpSettings).limit(1))[0]
if (!row?.enabled) return if (!row?.enabled) return
const root = String(row.baseUrl ?? "").replace(/\/+$/, "") const root = String(row.baseUrl ?? "").replace(/\/+$/, "")
const token = String(row.apiKey ?? "").replace(/^Bearer\s+/i, "").trim() const token = String(row.apiKey ?? "").replace(/^Bearer\s+/i, "").trim()
@@ -8,7 +8,6 @@ export interface ExporterMapPayload {
} }
export interface CollectorStartPayload { export interface CollectorStartPayload {
dbPath: string
listenHost: string listenHost: string
listenPort: number listenPort: number
topN: number topN: number
@@ -1,6 +1,5 @@
import { createSocket, type Socket } from "node:dgram" import { createSocket, type Socket } from "node:dgram"
import { parentPort } from "node:worker_threads" import { parentPort } from "node:worker_threads"
import { sqliteDatabase } from "../db/index.js"
import type { import type {
CollectorStartPayload, CollectorStartPayload,
ExporterMapPayload, ExporterMapPayload,
@@ -9,7 +8,6 @@ import type {
} from "./traffic-flow-collector-ipc.js" } from "./traffic-flow-collector-ipc.js"
import { import {
TICK_MS, TICK_MS,
attachEngineSqlite,
configureEngine, configureEngine,
flushPending, flushPending,
getEngineStats, getEngineStats,
@@ -23,7 +21,6 @@ let socket: Socket | null = null
let flushTimer: ReturnType<typeof setInterval> | null = null let flushTimer: ReturnType<typeof setInterval> | null = null
let bound = false let bound = false
let address: string | null = null let address: string | null = null
let attached = false
function send(msg: WorkerToMain): void { function send(msg: WorkerToMain): void {
parentPort?.postMessage(msg) parentPort?.postMessage(msg)
@@ -58,22 +55,14 @@ function applyExporterMap(payload: ExporterMapPayload): void {
}) })
} }
function ensureSqlite(): void {
if (attached) return
attachEngineSqlite(sqliteDatabase)
attached = true
}
function stopListener(): void { function stopListener(): void {
if (flushTimer) { if (flushTimer) {
clearInterval(flushTimer) clearInterval(flushTimer)
flushTimer = null flushTimer = null
} }
try { void flushPending().catch((e) => {
flushPending()
} catch (e) {
setEngineError(e instanceof Error ? e.message : String(e)) setEngineError(e instanceof Error ? e.message : String(e))
} })
if (socket) { if (socket) {
try { socket.close() } catch { /* ignore */ } try { socket.close() } catch { /* ignore */ }
socket = null socket = null
@@ -84,7 +73,6 @@ function stopListener(): void {
function startListener(payload: CollectorStartPayload): void { function startListener(payload: CollectorStartPayload): void {
stopListener() stopListener()
ensureSqlite()
configureEngine({ topN: payload.topN, retentionHours: payload.retentionHours }) configureEngine({ topN: payload.topN, retentionHours: payload.retentionHours })
applyExporterMap(payload.exporterMap) applyExporterMap(payload.exporterMap)
@@ -116,12 +104,9 @@ function startListener(payload: CollectorStartPayload): void {
}) })
socket = sock socket = sock
flushTimer = setInterval(() => { flushTimer = setInterval(() => {
try { void flushPending().catch((e) => {
flushPending()
} catch (e) {
setEngineError(e instanceof Error ? e.message : String(e)) setEngineError(e instanceof Error ? e.message : String(e))
} }).finally(() => heartbeat())
heartbeat()
}, TICK_MS) }, TICK_MS)
} }
+157 -174
View File
@@ -1,4 +1,5 @@
import type Database from "better-sqlite3" import { dbAll, dbQuery, pool } from "../db/index.js"
import { ensurePartitionFor, specForParent } from "../db/partitions.js"
import { normalizeParsedFlow, parseFlowPacket, protoName, type ParsedFlow, type ParsedFlowInput } from "./traffic-flow-parse.js" import { normalizeParsedFlow, parseFlowPacket, protoName, type ParsedFlow, type ParsedFlowInput } from "./traffic-flow-parse.js"
import { classifyFlowPlaneLite } from "./traffic-flow-planes.js" import { classifyFlowPlaneLite } from "./traffic-flow-planes.js"
import { pickServerIdForExporter, type OverlayPeerRef } from "./traffic-flow-map-exporter.js" import { pickServerIdForExporter, type OverlayPeerRef } from "./traffic-flow-map-exporter.js"
@@ -10,8 +11,6 @@ import { isIsoCountry } from "./traffic-flow-brands.js"
import { maybeRefreshIfaces } from "./traffic-flow-ifaces.js" import { maybeRefreshIfaces } from "./traffic-flow-ifaces.js"
import { pickInternetPeer } from "./traffic-flow-ip.js" import { pickInternetPeer } from "./traffic-flow-ip.js"
type SqliteHandle = InstanceType<typeof Database>
export const TICK_MS = 2_000 export const TICK_MS = 2_000
export const RING_LEN = 60 export const RING_LEN = 60
export const MAX_PENDING = 50_000 export const MAX_PENDING = 50_000
@@ -20,6 +19,17 @@ export const DAILY_RETENTION_DAYS = 396
export const MINUTE_RETENTION_HOURS = 48 export const MINUTE_RETENTION_HOURS = 48
let pendingCap = MAX_PENDING let pendingCap = MAX_PENDING
const ensuredParts = new Set<string>()
async function ensureParentPartition(parent: string, ts: string): Promise<void> {
const spec = specForParent(parent)
if (!spec) return
const iso = ts.length === 10 ? `${ts}T00:00:00Z` : ts
const key = `${parent}:${iso.slice(0, 10)}`
if (ensuredParts.has(key)) return
await ensurePartitionFor(pool, parent, spec.kind, new Date(iso))
ensuredParts.add(key)
}
export interface PendingFlowRow { export interface PendingFlowRow {
serverId: number serverId: number
@@ -76,7 +86,6 @@ export interface ExporterResolveCtx {
hostIps: Map<string, number> hostIps: Map<string, number>
} }
let sqliteRef: SqliteHandle | null = null
let topN = 200 let topN = 200
let retentionHours = 24 let retentionHours = 24
@@ -95,7 +104,7 @@ let dropped = 0
let rowsStored = 0 let rowsStored = 0
let lastFlushUsedTransaction = false let lastFlushUsedTransaction = false
let lastPruneAt = 0 let lastPruneAt = 0
let lastPassiveCheckpointAt = Date.now() let lastPassiveCheckpointAt = 0
let dataEpoch = 0 let dataEpoch = 0
let lastPersistedStats: { let lastPersistedStats: {
packetsReceived: number packetsReceived: number
@@ -107,7 +116,6 @@ let exporterCtx: ExporterResolveCtx | null = null
const PRUNE_MS = 5 * 60_000 const PRUNE_MS = 5 * 60_000
const LIVE_WINDOW_MS = 15 * 60_000 const LIVE_WINDOW_MS = 15 * 60_000
const PASSIVE_CHECKPOINT_MS = 60_000
function bumpDataEpoch(): void { function bumpDataEpoch(): void {
dataEpoch += 1 dataEpoch += 1
@@ -201,8 +209,8 @@ function bumpRollup(serverId: number, bucketAt: string, flow: ParsedFlow, bytes:
acc.conversations += 1 acc.conversations += 1
} }
export function attachEngineSqlite(handle: SqliteHandle): void { export function attachEngineSqlite(_handle?: unknown): void {
sqliteRef = handle /* engine uses pg pool via dbQuery/dbAll */
} }
export function setPendingCapForTests(n: number | null): void { export function setPendingCapForTests(n: number | null): void {
@@ -441,7 +449,7 @@ export function applyRingSnapshot(rows: Array<{ key: string; inBps: number[]; ou
} }
} }
function persistListenerStats(handle: SqliteHandle): boolean { async function persistListenerStats(): Promise<boolean> {
if ( if (
lastPersistedStats lastPersistedStats
&& lastPersistedStats.packetsReceived === packetsReceived && lastPersistedStats.packetsReceived === packetsReceived
@@ -451,7 +459,7 @@ function persistListenerStats(handle: SqliteHandle): boolean {
) { ) {
return false return false
} }
handle.prepare(` await dbQuery(`
UPDATE traffic_flow_settings UPDATE traffic_flow_settings
SET packets_received = @packetsReceived, SET packets_received = @packetsReceived,
last_datagram_at = @lastDatagramAt, last_datagram_at = @lastDatagramAt,
@@ -459,7 +467,7 @@ function persistListenerStats(handle: SqliteHandle): boolean {
last_error = @lastError, last_error = @lastError,
updated_at = @updatedAt updated_at = @updatedAt
WHERE id = 1 WHERE id = 1
`).run({ `, {
packetsReceived, packetsReceived,
lastDatagramAt, lastDatagramAt,
lastExporterIp, lastExporterIp,
@@ -476,110 +484,85 @@ function persistListenerStats(handle: SqliteHandle): boolean {
return true return true
} }
function maybePassiveCheckpoint(handle: SqliteHandle): void { async function upsertMinuteAndDaily(): Promise<void> {
const now = Date.now() for (const [k, acc] of minuteRollup) {
if (now - lastPassiveCheckpointAt < PASSIVE_CHECKPOINT_MS) return const [serverIdRaw, bucketAt] = k.split("\0")
lastPassiveCheckpointAt = now await ensureParentPartition("flow_minute_stats", bucketAt ?? "")
try { await dbQuery(`
handle.pragma("wal_checkpoint(PASSIVE)") INSERT INTO flow_minute_stats (
} catch { server_id, bucket_at, bytes, packets, unique_src, unique_dst, conversations
/* ignore */ ) VALUES (
@serverId, @bucketAt, @bytes, @packets, @uniqueSrc, @uniqueDst, @conversations
)
ON CONFLICT(server_id, bucket_at) DO UPDATE SET
bytes = flow_minute_stats.bytes + excluded.bytes,
packets = flow_minute_stats.packets + excluded.packets,
unique_src = GREATEST(flow_minute_stats.unique_src, excluded.unique_src),
unique_dst = GREATEST(flow_minute_stats.unique_dst, excluded.unique_dst),
conversations = flow_minute_stats.conversations + excluded.conversations
`, {
serverId: Number(serverIdRaw),
bucketAt,
bytes: acc.bytes,
packets: acc.packets,
uniqueSrc: acc.srcs.size,
uniqueDst: acc.dsts.size,
conversations: acc.conversations,
})
} }
} for (const [k, acc] of minuteDims) {
const [serverIdRaw, bucketAt, dim, key] = k.split("\0")
function upsertMinuteAndDaily(handle: SqliteHandle): void { await ensureParentPartition("flow_minute_dims", bucketAt ?? "")
const upsertMinute = handle.prepare(` await ensureParentPartition("flow_daily_dims", dayKey(bucketAt ?? ""))
INSERT INTO flow_minute_stats ( await dbQuery(`
server_id, bucket_at, bytes, packets, unique_src, unique_dst, conversations INSERT INTO flow_minute_dims (server_id, bucket_at, dim, key, bytes, packets)
) VALUES ( VALUES (@serverId, @bucketAt, @dim, @key, @bytes, @packets)
@serverId, @bucketAt, @bytes, @packets, @uniqueSrc, @uniqueDst, @conversations ON CONFLICT(server_id, bucket_at, dim, key) DO UPDATE SET
) bytes = flow_minute_dims.bytes + excluded.bytes,
ON CONFLICT(server_id, bucket_at) DO UPDATE SET packets = flow_minute_dims.packets + excluded.packets
bytes = bytes + excluded.bytes, `, {
packets = packets + excluded.packets, serverId: Number(serverIdRaw),
unique_src = MAX(unique_src, excluded.unique_src), bucketAt,
unique_dst = MAX(unique_dst, excluded.unique_dst), dim,
conversations = conversations + excluded.conversations key,
`) bytes: acc.bytes,
const upsertDim = handle.prepare(` packets: acc.packets,
INSERT INTO flow_minute_dims (server_id, bucket_at, dim, key, bytes, packets) })
VALUES (@serverId, @bucketAt, @dim, @key, @bytes, @packets) if (dim === "country" || dim === "service" || dim === "asn") {
ON CONFLICT(server_id, bucket_at, dim, key) DO UPDATE SET await dbQuery(`
bytes = bytes + excluded.bytes, INSERT INTO flow_daily_dims (server_id, day, dim, key, bytes, packets)
packets = packets + excluded.packets VALUES (@serverId, @day, @dim, @key, @bytes, @packets)
`) ON CONFLICT(server_id, day, dim, key) DO UPDATE SET
const upsertDaily = handle.prepare(` bytes = flow_daily_dims.bytes + excluded.bytes,
INSERT INTO flow_daily_dims (server_id, day, dim, key, bytes, packets) packets = flow_daily_dims.packets + excluded.packets
VALUES (@serverId, @day, @dim, @key, @bytes, @packets) `, {
ON CONFLICT(server_id, day, dim, key) DO UPDATE SET
bytes = bytes + excluded.bytes,
packets = packets + excluded.packets
`)
const tx = handle.transaction(() => {
for (const [k, acc] of minuteRollup) {
const [serverIdRaw, bucketAt] = k.split("\0")
upsertMinute.run({
serverId: Number(serverIdRaw), serverId: Number(serverIdRaw),
bucketAt, day: dayKey(bucketAt ?? ""),
bytes: acc.bytes,
packets: acc.packets,
uniqueSrc: acc.srcs.size,
uniqueDst: acc.dsts.size,
conversations: acc.conversations,
})
}
for (const [k, acc] of minuteDims) {
const [serverIdRaw, bucketAt, dim, key] = k.split("\0")
upsertDim.run({
serverId: Number(serverIdRaw),
bucketAt,
dim, dim,
key, key,
bytes: acc.bytes, bytes: acc.bytes,
packets: acc.packets, packets: acc.packets,
}) })
if (dim === "country" || dim === "service" || dim === "asn") {
upsertDaily.run({
serverId: Number(serverIdRaw),
day: dayKey(bucketAt ?? ""),
dim,
key,
bytes: acc.bytes,
packets: acc.packets,
})
}
} }
}) }
tx()
minuteRollup.clear() minuteRollup.clear()
minuteDims.clear() minuteDims.clear()
} }
function capDailyAsn(handle: SqliteHandle): void { async function capDailyAsn(): Promise<void> {
const today = nowIso().slice(0, 10) const today = nowIso().slice(0, 10)
const rows = handle.prepare(` const rows = await dbAll<{ serverId: number; key: string; bytes: number; packets: number }>(`
SELECT server_id AS serverId, key, bytes, packets SELECT server_id AS "serverId", key, bytes, packets
FROM flow_daily_dims FROM flow_daily_dims
WHERE day = ? AND dim = 'asn' WHERE day = ? AND dim = 'asn'
ORDER BY server_id, bytes DESC ORDER BY server_id, bytes DESC
`).all(today) as Array<{ serverId: number; key: string; bytes: number; packets: number }> `, [today])
const byServer = new Map<number, typeof rows>() const byServer = new Map<number, typeof rows>()
for (const row of rows) { for (const row of rows) {
const list = byServer.get(row.serverId) ?? [] const list = byServer.get(row.serverId) ?? []
list.push(row) list.push(row)
byServer.set(row.serverId, list) byServer.set(row.serverId, list)
} }
const del = handle.prepare(`
DELETE FROM flow_daily_dims WHERE server_id = ? AND day = ? AND dim = 'asn' AND key = ?
`)
const upsertOther = handle.prepare(`
INSERT INTO flow_daily_dims (server_id, day, dim, key, bytes, packets)
VALUES (?, ?, 'asn', 'other', ?, ?)
ON CONFLICT(server_id, day, dim, key) DO UPDATE SET
bytes = bytes + excluded.bytes,
packets = packets + excluded.packets
`)
for (const [serverId, list] of byServer) { for (const [serverId, list] of byServer) {
if (list.length <= DAILY_ASN_TOP) continue if (list.length <= DAILY_ASN_TOP) continue
let otherBytes = 0 let otherBytes = 0
@@ -588,52 +571,60 @@ function capDailyAsn(handle: SqliteHandle): void {
if (row.key === "other") continue if (row.key === "other") continue
otherBytes += row.bytes otherBytes += row.bytes
otherPackets += row.packets otherPackets += row.packets
del.run(serverId, today, row.key) await dbQuery(
`DELETE FROM flow_daily_dims WHERE server_id = ? AND day = ? AND dim = 'asn' AND key = ?`,
[serverId, today, row.key],
)
}
if (otherBytes > 0) {
await ensureParentPartition("flow_daily_dims", today)
await dbQuery(`
INSERT INTO flow_daily_dims (server_id, day, dim, key, bytes, packets)
VALUES (?, ?, 'asn', 'other', ?, ?)
ON CONFLICT(server_id, day, dim, key) DO UPDATE SET
bytes = flow_daily_dims.bytes + excluded.bytes,
packets = flow_daily_dims.packets + excluded.packets
`, [serverId, today, otherBytes, otherPackets])
} }
if (otherBytes > 0) upsertOther.run(serverId, today, otherBytes, otherPackets)
} }
} }
function pruneStored(handle: SqliteHandle): void { async function pruneStored(): Promise<void> {
const now = Date.now() const now = Date.now()
if (now - lastPruneAt < PRUNE_MS) return if (now - lastPruneAt < PRUNE_MS) return
lastPruneAt = now lastPruneAt = now
const flowCutoff = new Date(now - retentionHours * 3600_000).toISOString() const flowCutoff = new Date(now - retentionHours * 3600_000).toISOString()
const minuteCutoff = new Date(now - MINUTE_RETENTION_HOURS * 3600_000).toISOString() const minuteCutoff = new Date(now - MINUTE_RETENTION_HOURS * 3600_000).toISOString()
const dailyCutoff = new Date(now - DAILY_RETENTION_DAYS * 86400_000).toISOString().slice(0, 10) const dailyCutoff = new Date(now - DAILY_RETENTION_DAYS * 86400_000).toISOString().slice(0, 10)
handle.prepare(`DELETE FROM flow_buckets WHERE bucket_at < ?`).run(flowCutoff) await dbQuery(`DELETE FROM flow_buckets WHERE bucket_at < ?`, [flowCutoff])
handle.prepare(`DELETE FROM flow_minute_stats WHERE bucket_at < ?`).run(minuteCutoff) await dbQuery(`DELETE FROM flow_minute_stats WHERE bucket_at < ?`, [minuteCutoff])
handle.prepare(`DELETE FROM flow_minute_dims WHERE bucket_at < ?`).run(minuteCutoff) await dbQuery(`DELETE FROM flow_minute_dims WHERE bucket_at < ?`, [minuteCutoff])
handle.prepare(`DELETE FROM flow_daily_dims WHERE day < ?`).run(dailyCutoff) await dbQuery(`DELETE FROM flow_daily_dims WHERE day < ?`, [dailyCutoff])
const keep = Math.max(20, topN) const keep = Math.max(20, topN)
try { await dbQuery(`
handle.prepare(` DELETE FROM flow_buckets fb
DELETE FROM flow_buckets WHERE id IN ( USING (
SELECT id FROM ( SELECT server_id, bucket_at, src, dst, proto, src_port, dst_port, in_iface
SELECT id, ROW_NUMBER() OVER ( FROM (
PARTITION BY server_id, bucket_at ORDER BY bytes DESC SELECT server_id, bucket_at, src, dst, proto, src_port, dst_port, in_iface,
) AS rn ROW_NUMBER() OVER (
FROM flow_buckets PARTITION BY server_id, bucket_at ORDER BY bytes DESC
) ranked WHERE rn > ? ) AS rn
) FROM flow_buckets
`).run(keep) ) ranked
} catch { WHERE rn > ?
const buckets = handle.prepare(` ) drop_rows
SELECT DISTINCT server_id AS serverId, bucket_at AS bucketAt FROM flow_buckets WHERE fb.server_id = drop_rows.server_id
`).all() as Array<{ serverId: number; bucketAt: string }> AND fb.bucket_at = drop_rows.bucket_at
for (const b of buckets) { AND fb.src = drop_rows.src
const rows = handle.prepare(` AND fb.dst = drop_rows.dst
SELECT id, bytes FROM flow_buckets AND fb.proto = drop_rows.proto
WHERE server_id = ? AND bucket_at = ? AND fb.src_port = drop_rows.src_port
ORDER BY bytes DESC AND fb.dst_port = drop_rows.dst_port
`).all(b.serverId, b.bucketAt) as Array<{ id: number; bytes: number }> AND fb.in_iface = drop_rows.in_iface
for (const extra of rows.slice(keep)) { `, [keep])
handle.prepare(`DELETE FROM flow_buckets WHERE id = ?`).run(extra.id) await pruneRipeSqlite(now)
}
}
}
pruneRipeSqlite(now)
} }
function topNPending(rows: PendingFlowRow[]): PendingFlowRow[] { function topNPending(rows: PendingFlowRow[]): PendingFlowRow[] {
@@ -653,18 +644,12 @@ function topNPending(rows: PendingFlowRow[]): PendingFlowRow[] {
return out return out
} }
export function flushPending(): void { export async function flushPending(): Promise<void> {
pruneRecent() pruneRecent()
rollFlowRings() rollFlowRings()
const handle = sqliteRef await persistListenerStats()
if (!handle) {
lastFlushUsedTransaction = false
return
}
persistListenerStats(handle)
if (pending.size === 0 && minuteRollup.size === 0 && minuteDims.size === 0) { if (pending.size === 0 && minuteRollup.size === 0 && minuteDims.size === 0) {
pruneStored(handle) await pruneStored()
maybePassiveCheckpoint(handle)
lastFlushUsedTransaction = false lastFlushUsedTransaction = false
return return
} }
@@ -672,7 +657,7 @@ export function flushPending(): void {
pending.clear() pending.clear()
for (const row of rows) mergeInto(recent, row) for (const row of rows) mergeInto(recent, row)
const upsertFlow = handle.prepare(` const upsertSql = `
INSERT INTO flow_buckets ( INSERT INTO flow_buckets (
server_id, bucket_at, src, dst, proto, src_port, dst_port, bytes, packets, in_iface, out_iface, next_hop, flow_start_ms, flow_end_ms server_id, bucket_at, src, dst, proto, src_port, dst_port, bytes, packets, in_iface, out_iface, next_hop, flow_start_ms, flow_end_ms
) VALUES ( ) VALUES (
@@ -680,45 +665,44 @@ export function flushPending(): void {
) )
ON CONFLICT(server_id, bucket_at, src, dst, proto, src_port, dst_port, in_iface) ON CONFLICT(server_id, bucket_at, src, dst, proto, src_port, dst_port, in_iface)
DO UPDATE SET DO UPDATE SET
bytes = bytes + excluded.bytes, bytes = flow_buckets.bytes + excluded.bytes,
packets = packets + excluded.packets, packets = flow_buckets.packets + excluded.packets,
out_iface = CASE WHEN excluded.out_iface != '' THEN excluded.out_iface ELSE out_iface END, out_iface = CASE WHEN excluded.out_iface != '' THEN excluded.out_iface ELSE flow_buckets.out_iface END,
next_hop = CASE WHEN excluded.next_hop != '' THEN excluded.next_hop ELSE next_hop END, next_hop = CASE WHEN excluded.next_hop != '' THEN excluded.next_hop ELSE flow_buckets.next_hop END,
flow_start_ms = CASE flow_start_ms = CASE
WHEN excluded.flow_start_ms > 0 AND (flow_start_ms = 0 OR excluded.flow_start_ms < flow_start_ms) WHEN excluded.flow_start_ms > 0 AND (flow_buckets.flow_start_ms = 0 OR excluded.flow_start_ms < flow_buckets.flow_start_ms)
THEN excluded.flow_start_ms ELSE flow_start_ms END, THEN excluded.flow_start_ms ELSE flow_buckets.flow_start_ms END,
flow_end_ms = MAX(flow_end_ms, excluded.flow_end_ms) flow_end_ms = GREATEST(flow_buckets.flow_end_ms, excluded.flow_end_ms)
`) `
lastFlushUsedTransaction = false lastFlushUsedTransaction = false
try { try {
const tx = handle.transaction((batch: PendingFlowRow[]) => { for (const r of rows) {
for (const r of batch) { await ensureParentPartition("flow_buckets", r.bucketAt)
upsertFlow.run({ await dbQuery(upsertSql, {
serverId: r.serverId, serverId: r.serverId,
bucketAt: r.bucketAt, bucketAt: r.bucketAt,
src: r.src, src: r.src,
dst: r.dst, dst: r.dst,
proto: r.proto, proto: r.proto,
srcPort: r.srcPort, srcPort: r.srcPort,
dstPort: r.dstPort, dstPort: r.dstPort,
bytes: r.bytes, bytes: r.bytes,
packets: r.packets, packets: r.packets,
inIface: r.inIface, inIface: r.inIface,
outIface: r.outIface, outIface: r.outIface,
nextHop: r.nextHop, nextHop: r.nextHop,
flowStartMs: r.flowStartMs, flowStartMs: r.flowStartMs,
flowEndMs: r.flowEndMs, flowEndMs: r.flowEndMs,
}) })
} }
})
tx(rows)
lastFlushUsedTransaction = true lastFlushUsedTransaction = true
rowsStored += rows.length rowsStored += rows.length
bumpDataEpoch() bumpDataEpoch()
} catch { } catch {
for (const r of rows) { for (const r of rows) {
try { try {
upsertFlow.run({ await ensureParentPartition("flow_buckets", r.bucketAt)
await dbQuery(upsertSql, {
serverId: r.serverId, serverId: r.serverId,
bucketAt: r.bucketAt, bucketAt: r.bucketAt,
src: r.src, src: r.src,
@@ -741,25 +725,24 @@ export function flushPending(): void {
} }
} }
try { try {
upsertMinuteAndDaily(handle) await upsertMinuteAndDaily()
capDailyAsn(handle) await capDailyAsn()
} catch { } catch {
/* rollup best-effort */ /* rollup best-effort */
} }
pruneStored(handle) await pruneStored()
maybePassiveCheckpoint(handle)
} }
export function lastFlushUsedTransactionForTests(): boolean { export function lastFlushUsedTransactionForTests(): boolean {
return lastFlushUsedTransaction return lastFlushUsedTransaction
} }
export function flushPendingForTests(): void { export async function flushPendingForTests(): Promise<void> {
flushPending() await flushPending()
} }
export function onEngineTick(): void { export function onEngineTick(): void {
flushPending() void flushPending()
} }
export function ingestParsedFlowsForServerForTests(serverId: number, flows: ParsedFlowInput[]): void { export function ingestParsedFlowsForServerForTests(serverId: number, flows: ParsedFlowInput[]): void {
@@ -1,38 +0,0 @@
import assert from "node:assert/strict"
import {
SQLITE_BUSY_TIMEOUT_MS,
SQLITE_CACHE_SIZE_KIB,
SQLITE_WAL_AUTOCHECKPOINT_PAGES,
sqliteDatabase,
} from "../db/index.js"
import {
MAX_FLOW_LIVE_SUBSCRIBERS,
resetFlowLiveSlotsForTests,
tryAcquireFlowLiveSlot,
releaseFlowLiveSlot,
} from "../routes/traffic-flow.js"
const busy = sqliteDatabase.pragma("busy_timeout") as Array<{ busy_timeout: number }>
const busyValue = Array.isArray(busy) ? Number(Object.values(busy[0] ?? {})[0]) : Number(busy)
assert.equal(busyValue, SQLITE_BUSY_TIMEOUT_MS)
function pragmaNum(name: string): number {
const rows = sqliteDatabase.pragma(name) as Array<Record<string, number>>
const row = Array.isArray(rows) ? rows[0] : rows
return Number(Object.values(row ?? {})[0])
}
assert.equal(pragmaNum("wal_autocheckpoint"), SQLITE_WAL_AUTOCHECKPOINT_PAGES)
assert.equal(pragmaNum("cache_size"), -SQLITE_CACHE_SIZE_KIB)
assert.equal(pragmaNum("temp_store"), 2)
resetFlowLiveSlotsForTests()
for (let i = 0; i < MAX_FLOW_LIVE_SUBSCRIBERS; i++) {
assert.equal(tryAcquireFlowLiveSlot(), true)
}
assert.equal(tryAcquireFlowLiveSlot(), false)
releaseFlowLiveSlot()
assert.equal(tryAcquireFlowLiveSlot(), true)
resetFlowLiveSlotsForTests()
console.log("traffic-flow-hardening.test.ts: ok")
+13 -13
View File
@@ -4,9 +4,9 @@ import type { TrafficFlowHostFile } from "@mmapp/contracts/traffic-flow"
const COMPOSE_DIR = "/opt/cdn-mm" const COMPOSE_DIR = "/opt/cdn-mm"
export function buildHostWgQuickConf(): string { export async function buildHostWgQuickConf(): Promise<string> {
const row = getTrafficFlowSettingsRow() const row = await getTrafficFlowSettingsRow()
const peers = listHostPeers() const peers = await listHostPeers()
return generateNativeConf({ return generateNativeConf({
name: "wg-flow", name: "wg-flow",
mtu: 1420, mtu: 1420,
@@ -23,8 +23,8 @@ export function buildHostWgQuickConf(): string {
}) })
} }
export function buildHostComposeOverride(): string { export async function buildHostComposeOverride(): Promise<string> {
const row = getTrafficFlowSettingsRow() const row = await getTrafficFlowSettingsRow()
return [ return [
"# Docker Compose merge для /opt/cdn-mm", "# Docker Compose merge для /opt/cdn-mm",
"# Не править docker-compose.yml. Traefik не трогать.", "# Не править docker-compose.yml. Traefik не трогать.",
@@ -42,10 +42,10 @@ export function buildHostComposeOverride(): string {
].join("\n") ].join("\n")
} }
export function buildHostLinuxInstallSh(): string { export async function buildHostLinuxInstallSh(): Promise<string> {
const row = getTrafficFlowSettingsRow() const row = await getTrafficFlowSettingsRow()
const conf = buildHostWgQuickConf().replace(/\s+$/, "") + "\n" const conf = (await buildHostWgQuickConf()).replace(/\s+$/, "") + "\n"
const override = buildHostComposeOverride() const override = await buildHostComposeOverride()
const collector = row.collectorIp const collector = row.collectorIp
const flowPort = row.flowListenPort const flowPort = row.flowListenPort
@@ -104,10 +104,10 @@ echo "Готово. Traefik не трогали. UDP \${FLOW_PORT} только
` `
} }
export function listTrafficFlowHostFiles(): TrafficFlowHostFile[] { export async function listTrafficFlowHostFiles(): Promise<TrafficFlowHostFile[]> {
return [ return [
{ id: "linux", label: "Linux", filename: "install-wg-flow.sh", code: buildHostLinuxInstallSh() }, { id: "linux", label: "Linux", filename: "install-wg-flow.sh", code: await buildHostLinuxInstallSh() },
{ id: "wg-quick", label: "wg-flow.conf", filename: "wg-flow.conf", code: buildHostWgQuickConf() }, { id: "wg-quick", label: "wg-flow.conf", filename: "wg-flow.conf", code: await buildHostWgQuickConf() },
{ id: "compose", label: "compose", filename: "docker-compose.override.yml", code: buildHostComposeOverride() }, { id: "compose", label: "compose", filename: "docker-compose.override.yml", code: await buildHostComposeOverride() },
] ]
} }
+1 -1
View File
@@ -28,7 +28,7 @@ async function refreshServerIfacesInner(serverId: number, force = false): Promis
if (!force && !shouldRefreshIfaces(serverId)) return if (!force && !shouldRefreshIfaces(serverId)) return
inflight.add(serverId) inflight.add(serverId)
try { try {
const row = db.select().from(servers).where(eq(servers.id, serverId)).limit(1).all()[0] const row = (await db.select().from(servers).where(eq(servers.id, serverId)).limit(1))[0]
if (!row) return if (!row) return
const client = MikrotikClient.fromServer(row) const client = MikrotikClient.fromServer(row)
const ifaces = await client.get<RosIfaceIndexRow[]>("/interface") const ifaces = await client.get<RosIfaceIndexRow[]>("/interface")
@@ -22,7 +22,8 @@ import {
simulateWorkerExitForTests, simulateWorkerExitForTests,
} from "./traffic-flow-ingest.js" } from "./traffic-flow-ingest.js"
import { configureEngine, droppedForTests, pendingSizeForTests } from "./traffic-flow-engine.js" import { configureEngine, droppedForTests, pendingSizeForTests } from "./traffic-flow-engine.js"
import { sqliteDatabase } from "../db/index.js" import { dbQuery } from "../db/index.js"
import { withPgOrSkip } from "../test/pg.js"
resetIfaceCacheForTests() resetIfaceCacheForTests()
resetFlowRingsForTests() resetFlowRingsForTests()
@@ -81,16 +82,23 @@ const talkers = Array.from({ length: 25 }, (_, i) => ({
inIface: "2", inIface: "2",
outIface: "", outIface: "",
})) }))
ingestParsedFlowsForServerForTests(9, talkers) if (await withPgOrSkip()) {
flushPendingForTests() await dbQuery(`
const stored = sqliteDatabase.prepare(` INSERT INTO servers (id, name, host) VALUES (91009, 'flow-ingest-test', '127.0.0.1')
SELECT COUNT(*) AS n FROM flow_buckets WHERE server_id = 9 ON CONFLICT (id) DO NOTHING
`).get() as { n: number } `)
assert.ok(stored.n <= 20, `expected topN cap, got ${stored.n}`) ingestParsedFlowsForServerForTests(91009, talkers)
sqliteDatabase.prepare(`DELETE FROM flow_buckets WHERE server_id = 9`).run() await flushPendingForTests()
sqliteDatabase.prepare(`DELETE FROM flow_minute_stats WHERE server_id = 9`).run() const stored = await dbQuery<{ n: number }>(
sqliteDatabase.prepare(`DELETE FROM flow_minute_dims WHERE server_id = 9`).run() `SELECT COUNT(*)::int AS n FROM flow_buckets WHERE server_id = 91009`,
sqliteDatabase.prepare(`DELETE FROM flow_daily_dims WHERE server_id = 9`).run() )
assert.ok((stored.rows[0]?.n ?? 99) <= 20, `expected topN cap, got ${stored.rows[0]?.n}`)
await dbQuery(`DELETE FROM flow_buckets WHERE server_id = 91009`)
await dbQuery(`DELETE FROM flow_minute_stats WHERE server_id = 91009`)
await dbQuery(`DELETE FROM flow_minute_dims WHERE server_id = 91009`)
await dbQuery(`DELETE FROM flow_daily_dims WHERE server_id = 91009`)
await dbQuery(`DELETE FROM servers WHERE id = 91009`)
}
applyHeartbeatForTests({ applyHeartbeatForTests({
bound: true, bound: true,
@@ -106,7 +114,7 @@ applyHeartbeatForTests({
rings: [], rings: [],
}) })
assert.equal(getFlowListenerState().bound, true) assert.equal(getFlowListenerState().bound, true)
assert.equal(getFlowRuntimeCounters().packetsReceived, 42) assert.equal((await getFlowRuntimeCounters()).packetsReceived, 42)
assert.equal(getFlowWorkerHealth().alive, false) assert.equal(getFlowWorkerHealth().alive, false)
setWantListenForTests(true) setWantListenForTests(true)
assert.equal(simulateWorkerExitForTests(), 1) assert.equal(simulateWorkerExitForTests(), 1)
+93 -118
View File
@@ -1,15 +1,12 @@
import { Worker } from "node:worker_threads" import { Worker } from "node:worker_threads"
import { existsSync, statSync } from "node:fs"
import path from "node:path"
import { gte, sql } from "drizzle-orm" import { gte, sql } from "drizzle-orm"
import { beginSqliteExclusiveOp, db, endSqliteExclusiveOp, sqliteDatabase } from "../db/index.js" import { db, dbGet, dbQuery, pool, withAdvisoryLock } from "../db/index.js"
import { env } from "../config.js" import { dropExpiredPartitions } from "../db/partitions.js"
import { flowBuckets, servers } from "../db/schema.js" import { flowBuckets, servers } from "../db/schema.js"
import type { FlowPurgeDto, FlowStatsDto, FlowTalkerDto } from "@mmapp/contracts/traffic-flow" import type { FlowPurgeDto, FlowStatsDto, FlowTalkerDto } from "@mmapp/contracts/traffic-flow"
import { protoName, type ParsedFlowInput } from "./traffic-flow-parse.js" import { protoName, type ParsedFlowInput } from "./traffic-flow-parse.js"
import type { CollectorHeartbeat, ExporterMapPayload, MainToWorker, WorkerToMain } from "./traffic-flow-collector-ipc.js" import type { CollectorHeartbeat, ExporterMapPayload, MainToWorker, WorkerToMain } from "./traffic-flow-collector-ipc.js"
import { import {
attachEngineSqlite,
applyRingSnapshot, applyRingSnapshot,
configureEngine, configureEngine,
flushPending, flushPending,
@@ -57,8 +54,6 @@ let lastHeartbeat: CollectorHeartbeat | null = null
let state: FlowListenerState = { bound: false, address: null } let state: FlowListenerState = { bound: false, address: null }
let wantListen = false let wantListen = false
attachEngineSqlite(sqliteDatabase)
function workerFileUrl(): URL { function workerFileUrl(): URL {
const ts = import.meta.url.includes(".ts") const ts = import.meta.url.includes(".ts")
return new URL( return new URL(
@@ -67,13 +62,13 @@ function workerFileUrl(): URL {
) )
} }
export function buildExporterMapPayload(): ExporterMapPayload { export async function buildExporterMapPayload(): Promise<ExporterMapPayload> {
const settings = getTrafficFlowSettingsRow() const settings = await getTrafficFlowSettingsRow()
const rows = db.select({ const rows = await db.select({
id: servers.id, id: servers.id,
host: servers.host, host: servers.host,
mgmtTunnelIp: servers.mgmtTunnelIp, mgmtTunnelIp: servers.mgmtTunnelIp,
}).from(servers).all() }).from(servers)
const byTunnelIp: Array<[string, number]> = [] const byTunnelIp: Array<[string, number]> = []
const hostIps: Array<[string, number]> = [] const hostIps: Array<[string, number]> = []
for (const row of rows) { for (const row of rows) {
@@ -83,13 +78,13 @@ export function buildExporterMapPayload(): ExporterMapPayload {
return { return {
overlayPrefix: settings.prefix, overlayPrefix: settings.prefix,
byTunnelIp, byTunnelIp,
peers: listHostPeers(), peers: await listHostPeers(),
hostIps, hostIps,
} }
} }
function applyExporterCtxFromDb(): void { async function applyExporterCtxFromDb(): Promise<void> {
const payload = buildExporterMapPayload() const payload = await buildExporterMapPayload()
setExporterResolveCtx({ setExporterResolveCtx({
overlayPrefix: payload.overlayPrefix, overlayPrefix: payload.overlayPrefix,
byTunnelIp: new Map(payload.byTunnelIp), byTunnelIp: new Map(payload.byTunnelIp),
@@ -117,11 +112,11 @@ function handleWorkerMessage(msg: WorkerToMain): void {
} }
} }
function spawnWorker(): void { async function spawnWorker(): Promise<void> {
stopWorkerProcess() stopWorkerProcess()
const settings = getTrafficFlowSettingsRow() const settings = await getTrafficFlowSettingsRow()
configureEngine({ topN: settings.topN, retentionHours: settings.retentionHours }) configureEngine({ topN: settings.topN, retentionHours: settings.retentionHours })
applyExporterCtxFromDb() await applyExporterCtxFromDb()
const w = new Worker(workerFileUrl(), { execArgv: process.execArgv }) const w = new Worker(workerFileUrl(), { execArgv: process.execArgv })
w.on("message", (msg: WorkerToMain) => handleWorkerMessage(msg)) w.on("message", (msg: WorkerToMain) => handleWorkerMessage(msg))
w.on("error", (err) => { w.on("error", (err) => {
@@ -149,7 +144,7 @@ function spawnWorker(): void {
const delay = Math.min(30_000, 1000 * 2 ** restartAttempts) const delay = Math.min(30_000, 1000 * 2 ** restartAttempts)
restartAttempts += 1 restartAttempts += 1
restartTimer = setTimeout(() => { restartTimer = setTimeout(() => {
if (wantListen) spawnWorker() if (wantListen) void spawnWorker()
}, delay) }, delay)
void code void code
}) })
@@ -158,12 +153,11 @@ function spawnWorker(): void {
postToWorker({ postToWorker({
type: "start", type: "start",
payload: { payload: {
dbPath: env.DATABASE_PATH,
listenHost: host, listenHost: host,
listenPort: settings.flowListenPort, listenPort: settings.flowListenPort,
topN: settings.topN, topN: settings.topN,
retentionHours: settings.retentionHours, retentionHours: settings.retentionHours,
exporterMap: buildExporterMapPayload(), exporterMap: await buildExporterMapPayload(),
}, },
}) })
} }
@@ -184,8 +178,24 @@ function stopWorkerProcess(): void {
} }
} }
async function stopWorkerProcessAsync(): Promise<void> {
if (restartTimer) {
clearTimeout(restartTimer)
restartTimer = null
}
if (!worker) return
const current = worker
worker = null
try {
current.postMessage({ type: "stop" })
await current.terminate()
} catch {
/* ignore */
}
}
export function reattachFlowSqlite(): void { export function reattachFlowSqlite(): void {
attachEngineSqlite(sqliteDatabase) /* no-op: engine uses PostgreSQL pool */
} }
export function applyHeartbeatForTests(payload: CollectorHeartbeat): void { export function applyHeartbeatForTests(payload: CollectorHeartbeat): void {
@@ -221,44 +231,49 @@ export function getFlowWorkerHealth(): FlowWorkerHealth {
} }
} }
export function getFlowRuntimeCounters() { export async function getFlowRuntimeCounters() {
const settings = getTrafficFlowSettingsRow()
const hb = lastHeartbeat const hb = lastHeartbeat
if (hb) {
return {
packetsReceived: hb.packetsReceived,
lastExporterIp: hb.lastExporterIp ?? null,
lastError: hb.lastError || null,
lastDatagramAt: hb.lastDatagramAt ?? null,
dropped: hb.dropped ?? 0,
}
}
const settings = await getTrafficFlowSettingsRow()
return { return {
packetsReceived: hb?.packetsReceived ?? settings.packetsReceived, packetsReceived: settings.packetsReceived,
lastExporterIp: hb?.lastExporterIp ?? settings.lastExporterIp ?? null, lastExporterIp: settings.lastExporterIp ?? null,
lastError: (hb?.lastError ?? settings.lastError) || null, lastError: settings.lastError || null,
lastDatagramAt: hb?.lastDatagramAt ?? settings.lastDatagramAt ?? null, lastDatagramAt: settings.lastDatagramAt ?? null,
dropped: hb?.dropped ?? 0, dropped: 0,
} }
} }
export function startTrafficFlowListener() { export async function startTrafficFlowListener() {
stopTrafficFlowListener() stopTrafficFlowListener()
const settings = getTrafficFlowSettingsRow() const settings = await getTrafficFlowSettingsRow()
if (!settings.enabled) { if (!settings.enabled) {
wantListen = false wantListen = false
state = { bound: false, address: null } state = { bound: false, address: null }
return return
} }
wantListen = true wantListen = true
spawnWorker() await spawnWorker()
} }
export function stopTrafficFlowListener() { export function stopTrafficFlowListener() {
wantListen = false wantListen = false
stopWorkerProcess() stopWorkerProcess()
try { void flushPending().catch(() => { /* ignore */ })
flushPending()
} catch {
/* ignore */
}
state = { bound: false, address: null } state = { bound: false, address: null }
} }
export function refreshFlowExporterMap(): void { export async function refreshFlowExporterMap(): Promise<void> {
applyExporterCtxFromDb() await applyExporterCtxFromDb()
postToWorker({ type: "updateExporterMap", payload: buildExporterMapPayload() }) postToWorker({ type: "updateExporterMap", payload: await buildExporterMapPayload() })
} }
export function getRingMbps(serverId: number, iface = "__all__") { export function getRingMbps(serverId: number, iface = "__all__") {
@@ -280,21 +295,20 @@ function mergeInto(map: Map<string, PendingFlowRow>, row: PendingFlowRow): void
map.set(key, { ...row }) map.set(key, { ...row })
} }
export function listLiveFlowRows(sinceIso: string): PendingFlowRow[] { export async function listLiveFlowRows(sinceIso: string): Promise<PendingFlowRow[]> {
if (worker && lastHeartbeat?.workerAlive) { if (worker && lastHeartbeat?.workerAlive) {
return listStoredFlowRows(sinceIso) return await listStoredFlowRows(sinceIso)
} }
return engineListLive(sinceIso) return engineListLive(sinceIso)
} }
export function listStoredFlowRows(sinceIso: string): PendingFlowRow[] { export async function listStoredFlowRows(sinceIso: string): Promise<PendingFlowRow[]> {
const settings = getTrafficFlowSettingsRow() const settings = await getTrafficFlowSettingsRow()
const cap = Math.max(20, settings.topN) * 60 const cap = Math.max(20, settings.topN) * 60
const stored = db.select().from(flowBuckets) const stored = await db.select().from(flowBuckets)
.where(gte(flowBuckets.bucketAt, sinceIso)) .where(gte(flowBuckets.bucketAt, sinceIso))
.orderBy(sql`${flowBuckets.bytes} DESC`) .orderBy(sql`${flowBuckets.bytes} DESC`)
.limit(cap) .limit(cap)
.all()
const merged = new Map<string, PendingFlowRow>() const merged = new Map<string, PendingFlowRow>()
for (const r of stored) { for (const r of stored) {
mergeInto(merged, { mergeInto(merged, {
@@ -323,17 +337,17 @@ export function listStoredFlowRows(sinceIso: string): PendingFlowRow[] {
return [...merged.values()] return [...merged.values()]
} }
export function listFlowRowsForWindow(minutes: number): PendingFlowRow[] { export async function listFlowRowsForWindow(minutes: number): Promise<PendingFlowRow[]> {
const sinceIso = new Date(Date.now() - minutes * 60_000).toISOString() const sinceIso = new Date(Date.now() - minutes * 60_000).toISOString()
if (minutes <= 15 && !worker) return listLiveFlowRows(sinceIso) if (minutes <= 15 && !worker) return await listLiveFlowRows(sinceIso)
return listStoredFlowRows(sinceIso) return await listStoredFlowRows(sinceIso)
} }
export function listFlowTalkers(minutes = 5): FlowStatsDto { export async function listFlowTalkers(minutes = 5): Promise<FlowStatsDto> {
const settings = getTrafficFlowSettingsRow() const settings = await getTrafficFlowSettingsRow()
const runtime = getFlowRuntimeCounters() const runtime = await getFlowRuntimeCounters()
const rows = listFlowRowsForWindow(minutes) const rows = await listFlowRowsForWindow(minutes)
const catalog = getServerCatalog() const catalog = await getServerCatalog()
const nameById = new Map([...catalog.byId].map(([id, s]) => [id, s.name])) const nameById = new Map([...catalog.byId].map(([id, s]) => [id, s.name]))
const agg = new Map<string, FlowTalkerDto & { rawBytes: number }>() const agg = new Map<string, FlowTalkerDto & { rawBytes: number }>()
const protoBytes = new Map<number, number>() const protoBytes = new Map<number, number>()
@@ -405,13 +419,13 @@ export function listFlowTalkers(minutes = 5): FlowStatsDto {
} }
} }
export function ingestParsedFlowsForTests(exporterIp: string, flows: ParsedFlowInput[]) { export async function ingestParsedFlowsForTests(exporterIp: string, flows: ParsedFlowInput[]) {
applyExporterCtxFromDb() await applyExporterCtxFromDb()
const serverId = resolveServerId(exporterIp) const serverId = resolveServerId(exporterIp)
if (serverId == null) return if (serverId == null) return
queueParsedFlows(serverId, flows) queueParsedFlows(serverId, flows)
rollFlowRings() rollFlowRings()
flushPending() await flushPending()
} }
export function ingestParsedFlowsForServerForTests(serverId: number, flows: ParsedFlowInput[]) { export function ingestParsedFlowsForServerForTests(serverId: number, flows: ParsedFlowInput[]) {
@@ -420,7 +434,6 @@ export function ingestParsedFlowsForServerForTests(serverId: number, flows: Pars
export function resetFlowRingsForTests() { export function resetFlowRingsForTests() {
resetEngineForTests() resetEngineForTests()
attachEngineSqlite(sqliteDatabase)
lastHeartbeat = null lastHeartbeat = null
wantListen = false wantListen = false
restartAttempts = 0 restartAttempts = 0
@@ -430,88 +443,50 @@ export function lastFlushUsedTransactionForTests(): boolean {
return engineLastFlushTx() return engineLastFlushTx()
} }
export function flushPendingForTests(): void { export async function flushPendingForTests(): Promise<void> {
flushPending() await flushPending()
} }
function tableCount(name: string): number { async function tableCount(name: string): Promise<number> {
const row = sqliteDatabase.prepare(`SELECT COUNT(*) AS n FROM ${name}`).get() as { n: number } const row = await dbGet<{ n: number }>(`SELECT COUNT(*) AS n FROM ${name}`)
return Number(row?.n) || 0 return Number(row?.n) || 0
} }
function dbFileBytes(): number { const PURGE_LOCK_KEY = 8_723_103
const resolved = path.resolve(process.cwd(), env.DATABASE_PATH)
if (!existsSync(resolved)) return 0
return statSync(resolved).size
}
async function stopWorkerProcessAsync(): Promise<void> { /** Удаляет сессии и minute/daily rollup. Ключи WG и пиры JH не трогает. */
if (restartTimer) {
clearTimeout(restartTimer)
restartTimer = null
}
if (!worker) return
const current = worker
worker = null
try {
current.postMessage({ type: "stop" })
await current.terminate()
} catch {
/* ignore */
}
}
/** Удаляет сессии, minute/daily rollup и сжимает SQLite. Ключи WG и пиры JH не трогает. */
export async function purgeTrafficFlowStore(): Promise<FlowPurgeDto> { export async function purgeTrafficFlowStore(): Promise<FlowPurgeDto> {
beginSqliteExclusiveOp() return await withAdvisoryLock(PURGE_LOCK_KEY, async () => {
try {
wantListen = false wantListen = false
await stopWorkerProcessAsync() await stopWorkerProcessAsync()
resetEngineForTests() resetEngineForTests()
attachEngineSqlite(sqliteDatabase)
lastHeartbeat = null lastHeartbeat = null
state = { bound: false, address: null } state = { bound: false, address: null }
const fileBytesBefore = dbFileBytes()
const deleted = { const deleted = {
buckets: tableCount("flow_buckets"), buckets: await tableCount("flow_buckets"),
minuteStats: tableCount("flow_minute_stats"), minuteStats: await tableCount("flow_minute_stats"),
minuteDims: tableCount("flow_minute_dims"), minuteDims: await tableCount("flow_minute_dims"),
dailyDims: tableCount("flow_daily_dims"), dailyDims: await tableCount("flow_daily_dims"),
} }
sqliteDatabase.exec(` await dbQuery(`DELETE FROM flow_buckets`)
DELETE FROM flow_buckets; await dbQuery(`DELETE FROM flow_minute_stats`)
DELETE FROM flow_minute_stats; await dbQuery(`DELETE FROM flow_minute_dims`)
DELETE FROM flow_minute_dims; await dbQuery(`DELETE FROM flow_daily_dims`)
DELETE FROM flow_daily_dims; await resetFlowIngestCounters()
`) await dropExpiredPartitions(pool)
resetFlowIngestCounters()
try { try {
sqliteDatabase.pragma("wal_checkpoint(TRUNCATE)") await startTrafficFlowListener()
} catch { } catch {
/* ignore */ /* ingest мог остаться выключенным */
}
let vacuumed = false
try {
sqliteDatabase.exec("VACUUM")
vacuumed = true
} catch {
vacuumed = false
} }
return { return {
ok: true, ok: true,
deleted, deleted,
fileBytesBefore, fileBytesBefore: 0,
fileBytesAfter: dbFileBytes(), fileBytesAfter: 0,
vacuumed, vacuumed: false,
} }
} finally { })
try {
startTrafficFlowListener()
} catch {
/* ingest мог остаться выключенным */
}
endSqliteExclusiveOp()
}
} }
export { peekPendingFlows } export { peekPendingFlows }
@@ -5,6 +5,7 @@ import {
resetFlowRingsForTests, resetFlowRingsForTests,
} from "./traffic-flow-ingest.js" } from "./traffic-flow-ingest.js"
import { buildFlowMapHops, resetFlowMapHopsCacheForTests } from "./traffic-flow-map-hops.js" import { buildFlowMapHops, resetFlowMapHopsCacheForTests } from "./traffic-flow-map-hops.js"
import { withPgOrSkip } from "../test/pg.js"
import { seedFlowTopologyForTests, type FlowTopology } from "./traffic-flow-topology.js" import { seedFlowTopologyForTests, type FlowTopology } from "./traffic-flow-topology.js"
import { disableCatalogFetchForTests, resetFlowCatalogForTests } from "./traffic-flow-classify.js" import { disableCatalogFetchForTests, resetFlowCatalogForTests } from "./traffic-flow-classify.js"
import { import {
@@ -14,6 +15,11 @@ import {
seedRipeCacheForTests, seedRipeCacheForTests,
} from "./traffic-flow-ripe.js" } from "./traffic-flow-ripe.js"
if (!(await withPgOrSkip())) {
console.log("traffic-flow-map-hops.test.ts: skip")
process.exit(0)
}
disableCatalogFetchForTests() disableCatalogFetchForTests()
resetFlowCatalogForTests() resetFlowCatalogForTests()
disableRipePersistForTests() disableRipePersistForTests()
@@ -115,7 +121,7 @@ ingestParsedFlowsForServerForTests(3, [
try { try {
resetFlowMapHopsCacheForTests() resetFlowMapHopsCacheForTests()
const def = buildFlowMapHops({ minutes: 5 }) const def = await buildFlowMapHops({ minutes: 5 })
assert.equal(def.excludeOverlayApplied, true) assert.equal(def.excludeOverlayApplied, true)
assert.equal(def.excludeMeshApplied, true) assert.equal(def.excludeMeshApplied, true)
assert.equal(def.dedupApplied, true) assert.equal(def.dedupApplied, true)
@@ -144,7 +150,7 @@ try {
assert.equal(wan.bytes, 3000) assert.equal(wan.bytes, 3000)
resetFlowMapHopsCacheForTests() resetFlowMapHopsCacheForTests()
const withAll = buildFlowMapHops({ minutes: 5, excludeOverlay: false, excludeMesh: false }) const withAll = await buildFlowMapHops({ minutes: 5, excludeOverlay: false, excludeMesh: false })
const overlayIface = withAll.hops.find((h) => h.iface === "gre-jh-en" && h.fromId === "7") const overlayIface = withAll.hops.find((h) => h.iface === "gre-jh-en" && h.fromId === "7")
assert.ok(overlayIface && overlayIface.bytes >= 5_000_000) assert.ok(overlayIface && overlayIface.bytes >= 5_000_000)
const meshIface = withAll.hops.find((h) => h.iface === "gre-client" && h.fromId === "7" && h.kind === "iface") const meshIface = withAll.hops.find((h) => h.iface === "gre-client" && h.fromId === "7" && h.kind === "iface")
@@ -203,7 +209,7 @@ ingestParsedFlowsForServerForTests(7, [
]) ])
try { try {
resetFlowMapHopsCacheForTests() resetFlowMapHopsCacheForTests()
const six = buildFlowMapHops({ minutes: 5, minSharePct: 5 }) const six = await buildFlowMapHops({ minutes: 5, minSharePct: 5 })
assert.equal(six.totalBytes, 10_000) assert.equal(six.totalBytes, 10_000)
const google = six.services?.find((s) => s.id === "svc:google") const google = six.services?.find((s) => s.id === "svc:google")
assert.ok(google, "Google ≥ 5%") assert.ok(google, "Google ≥ 5%")
@@ -233,11 +239,11 @@ ingestParsedFlowsForServerForTests(7, [
]) ])
try { try {
resetFlowMapHopsCacheForTests() resetFlowMapHopsCacheForTests()
const four = buildFlowMapHops({ minutes: 5, minSharePct: 5 }) const four = await buildFlowMapHops({ minutes: 5, minSharePct: 5 })
assert.equal(four.totalBytes, 10_000) assert.equal(four.totalBytes, 10_000)
assert.ok(!(four.services ?? []).some((s) => s.id === "svc:google"), "Google < 5% hidden") assert.ok(!(four.services ?? []).some((s) => s.id === "svc:google"), "Google < 5% hidden")
resetFlowMapHopsCacheForTests() resetFlowMapHopsCacheForTests()
const off = buildFlowMapHops({ minutes: 5, minSharePct: 0 }) const off = await buildFlowMapHops({ minutes: 5, minSharePct: 0 })
assert.ok(off.services?.some((s) => s.id === "svc:google"), "порог 0 показывает Google 4%") assert.ok(off.services?.some((s) => s.id === "svc:google"), "порог 0 показывает Google 4%")
} finally { } finally {
resetFlowRingsForTests() resetFlowRingsForTests()
@@ -270,7 +276,7 @@ ingestParsedFlowsForServerForTests(7, [
]) ])
try { try {
resetFlowMapHopsCacheForTests() resetFlowMapHopsCacheForTests()
const greOnly = buildFlowMapHops({ minutes: 5, excludeOverlay: false, minSharePct: 0 }) const greOnly = await buildFlowMapHops({ minutes: 5, excludeOverlay: false, minSharePct: 0 })
assert.ok(!(greOnly.services ?? []).some((s) => s.label === "GRE"), "GRE is not a destination service") assert.ok(!(greOnly.services ?? []).some((s) => s.label === "GRE"), "GRE is not a destination service")
} finally { } finally {
seedFlowTopologyForTests(null) seedFlowTopologyForTests(null)
@@ -317,7 +323,7 @@ ingestParsedFlowsForServerForTests(7, [
]) ])
try { try {
resetFlowMapHopsCacheForTests() resetFlowMapHopsCacheForTests()
const rev = buildFlowMapHops({ minutes: 5, minSharePct: 0 }) const rev = await buildFlowMapHops({ minutes: 5, minSharePct: 0 })
assert.ok(rev.services?.some((s) => s.id === "svc:google"), "реверс Google:443 → 10.x") assert.ok(rev.services?.some((s) => s.id === "svc:google"), "реверс Google:443 → 10.x")
assert.ok(rev.services?.some((s) => s.id === "svc:cloudflare"), "реверс Cloudflare:443 → 10.x") assert.ok(rev.services?.some((s) => s.id === "svc:cloudflare"), "реверс Cloudflare:443 → 10.x")
assert.ok(rev.serviceEdges?.some((e) => e.toId === "svc:google" && e.fromId === "9")) assert.ok(rev.serviceEdges?.some((e) => e.toId === "svc:google" && e.fromId === "9"))
@@ -356,7 +362,7 @@ ingestParsedFlowsForServerForTests(7, [
]) ])
try { try {
resetFlowMapHopsCacheForTests() resetFlowMapHopsCacheForTests()
const wan = buildFlowMapHops({ minutes: 5, minSharePct: 0 }) const wan = await buildFlowMapHops({ minutes: 5, minSharePct: 0 })
const googleEdge = wan.serviceEdges?.find((e) => e.toId === "svc:google") const googleEdge = wan.serviceEdges?.find((e) => e.toId === "svc:google")
assert.ok(googleEdge, "Google с WAN JH") assert.ok(googleEdge, "Google с WAN JH")
assert.equal(googleEdge.fromId, "9", "якорь на EN, не на JH") assert.equal(googleEdge.fromId, "9", "якорь на EN, не на JH")
@@ -395,7 +401,7 @@ ingestParsedFlowsForServerForTests(7, [
]) ])
try { try {
resetFlowMapHopsCacheForTests() resetFlowMapHopsCacheForTests()
const wanOnly = buildFlowMapHops({ minutes: 5, minSharePct: 0 }) const wanOnly = await buildFlowMapHops({ minutes: 5, minSharePct: 0 })
const googleEdge = wanOnly.serviceEdges?.find((e) => e.toId === "svc:google") const googleEdge = wanOnly.serviceEdges?.find((e) => e.toId === "svc:google")
assert.ok(googleEdge, "Google WAN без GRE payload") assert.ok(googleEdge, "Google WAN без GRE payload")
assert.equal(googleEdge.fromId, "9", "единственный EN, даже без nextHop") assert.equal(googleEdge.fromId, "9", "единственный EN, даже без nextHop")

Some files were not shown because too many files have changed in this diff Show More