Compare commits

..
3 Commits
Author SHA1 Message Date
DenozordecandCursor 97e43b2335 feat(statistics): enhance interface handling and data aggregation
Docker images / prepare-release (push) Successful in 9s
Docker images / backend-test (push) Successful in 2m29s
Docker images / frontend-image (push) Successful in 3m24s
Docker images / updater-image (push) Successful in 44s
Docker images / backend-image (push) Successful in 2m39s
Docker images / notify-webhook (push) Skipped
Docker images / publish-release (push) Successful in 15s
Updated the statistics aggregation service to improve interface resolution and data handling. Introduced new functions for managing interface aliases and collapsing server interface rows, ensuring accurate data representation. Enhanced test coverage for interface resolution and added checks for new functionality.

- Implemented `factIfaceAliases` and `collapseServerIfaceRows` for better interface data management.
- Updated `resolveIfaceName` to handle additional cases for interface indexing.
- Enhanced tests for interface resolution and aggregation logic.

Co-authored-by: Cursor <[email protected]>
2026-09-10 21:49:50 +07:00
DenozordecandCursor db21e1217c feat(network-map): enhance service selection logic and UI updates
Docker images / prepare-release (push) Successful in 9s
Docker images / backend-test (push) Successful in 2m19s
Docker images / frontend-image (push) Successful in 2m52s
Docker images / updater-image (push) Successful in 46s
Docker images / backend-image (push) Successful in 2m20s
Docker images / notify-webhook (push) Skipped
Docker images / publish-release (push) Successful in 11s
Refactor the service selection mechanism to utilize a live reference for selected services, ensuring accurate display of service details. Update UI components to reflect changes in service selection, including dynamic updates for service metrics and paths. Additionally, introduce new utility functions for deduplicating flow rows across exporters in the backend, improving data handling in traffic flow analysis.

- Updated service selection logic in NetworkMapPage component.
- Enhanced UI to display live service data.
- Added deduplication functions in traffic-flow-dedup module for improved data integrity.

Co-authored-by: Cursor <[email protected]>
2026-09-10 19:54:16 +07:00
DenozordecandCursor 5bb9066be8 feat(statistics): добавить BI-разрез и сводную матрицу
Docker images / prepare-release (push) Successful in 10s
Docker images / backend-test (push) Successful in 2m23s
Docker images / frontend-image (push) Successful in 3m30s
Docker images / updater-image (push) Successful in 52s
Docker images / backend-image (push) Successful in 3m8s
Docker images / notify-webhook (push) Skipped
Docker images / publish-release (push) Successful in 14s
Сопоставить клиентов с ifIndex как на карте трафика, чтобы KPI пользователей не обнулялся. На экране — разрез остальных измерений и сводная матрица.

Co-authored-by: Cursor <[email protected]>
2026-09-10 16:07:25 +07:00
24 changed files with 1555 additions and 248 deletions
@@ -0,0 +1,63 @@
# Локальные GeoLite2-базы (Country + ASN) для потоков по странам и ASN
## Контекст
Сейчас страна и ASN для netflow-потоков резолвятся через внешний RIPEstat API (`backend/src/services/traffic-flow-ripe.ts`): лимит 30 новых префиксов/мин, очередь на 90, кэш в PG `flow_ip_meta`. Новые IP «дозревают» с задержкой, IPv6 не покрывается (кэш индексируется только по IPv4). Локальные mmdb-базы дают мгновенный синхронный lookup всех IP без внешних вызовов.
Решения (подтверждены):
- Источник — **MaxMind GeoLite2 через P3TERX-зеркало**: `https://github.com/P3TERX/GeoLite.mmdb/raw/download/GeoLite2-Country.mmdb` и `.../GeoLite2-ASN.mmdb`. Без регистрации, ключей и tar-распаковки. Точность по стране у GeoLite2 и IPinfo паритетная (<1% ошибок у обоих, arXiv 2026); выбран P3TERX за надёжность зеркала (5.2k звёзд) и преемственность: текущий RIPE-путь и так читает GeoLite (`maxmind-geo-lite`), история в кэше остаётся консистентной.
- **RIPEstat остаётся fallback** (до первой загрузки баз / если lookup не дал результата).
- City-базу не качаем (lat/lng фронтенд не использует).
## Изменения
### 1. Зависимость
- `npm install -w mikrotik-manager-backend maxmind` — sync-чтение mmdb, встроенные TS-типы, без транзитивных зависимостей, Node 22 ок.
### 2. Новый сервис `backend/src/services/traffic-flow-geoip.ts`
(по конвенциям окружения traffic-flow-*: контракт → маршрут → сервис, без БД-логики в маршрутах)
- Каталог: `backend/storage/geoip/` (конвенция `storage/backups`), файлы `GeoLite2-Country.mmdb`, `GeoLite2-ASN.mmdb`.
- `initGeoip()` — открыть ридеры best-effort при старте (из `index.ts` рядом с `startTrafficFlowListener`), независимо от настроек автообновления: файлы есть — работают.
- `lookupGeoip(ip): FlowIpMeta | null` — синхронно: `country.iso_code` (fallback `registered_country.iso_code`) с валидацией `isIsoCountry`, ASN = `autonomous_system_number`, holder = `autonomous_system_organization`; приватные IP → negative-запись как в RIPE (`isNonPublicIp`); в PG не пишем (lookup и так быстрый). IPv6 поддержан ридером.
- `resolveFlowIp(ip)` — фасад: `lookupGeoip(ip) ?? lookupRipeCached(ip)`; главный экспорт для потребителей.
- `geoipStatus()` — loaded, даты сборки баз (метаданные mmdb). Тест-хук `setGeoipReadersForTests`. Смена ридеров после обновления — атомарная замена ссылок.
### 3. Коллектор `backend/src/services/geoip-update-collector.ts`
`collectGeoipUpdateOnce()` по образцу `certificate-renew-collector.ts`:
1. Conditional GET с ETag/If-None-Match из настроек → 304 = skip (фолбэк-сравнение: размер/содержимое).
2. Скачивание в `*.tmp` через глобальный `fetch` + AbortController с таймаутом (внешний HTTP из service-слоя — по правилу fastify-backend-drizzle).
3. Валидация: открыть ридер из tmp-файла, пробой 8.8.8.8 (страна US, ASN 15169).
4. `fs.rename` атомарная подмена, старый файл → `*.prev` (откат, если новый ридер не открылся).
5. Перезагрузка ридеров, статус в настройках; snapshot для `scheduler_runs` (checked/downloaded/skipped/bytes/error).
### 4. Планировщик (`backend/src/services/scheduler.ts`)
- `JOB_KEYS` += `geoip_update`; case в `runSchedulerJobBody`; блок в `refreshScheduler()` по образцу `certificates_renew`: интервал `Math.max(6ч, updateIntervalSec*1000)`, по умолчанию 7 дней (upstream обновляется еженедельно) + немедленный первый запуск при включённой настройке.
### 5. Схема и миграция
- `backend/src/db/schema.ts`: singleton `geoip_settings``enabled` (default true), `updateIntervalSec` (default 604800), `lastCheckAt`, `lastSuccessAt`, `lastError`, `countryBuildAt`, `asnBuildAt`, `etagsJson` (jsonb), `createdAt/updatedAt`.
- Миграция: `npm run db:generate` → файл в `backend/drizzle/`.
### 6. API + контракты
- `packages/contracts/src/geoip.ts`: zod-схемы настроек/статуса (все входы — Zod, по правилам проекта).
- Новый `backend/src/routes/geoip.ts`, регистрация в `index.ts` с prefix `/api`:
- `GET /api/geoip` — настройки + статус (ready, даты сборки, последняя проверка/ошибка);
- `PUT /api/geoip` — сохранить настройки, затем `refreshScheduler()`;
- `POST /api/geoip/update` — запустить загрузку сейчас (409, если уже идёт; флаг-гард как в коллекторах).
### 7. Интеграция в пайплайн (geoip-first, RIPE-fallback)
- `traffic-flow-engine.ts` (`queueParsedFlows`, ~строка 336): `lookupRipeCached``resolveFlowIp`. Логика misses не меняется: при готовом mmdb публичные IP (v4+v6) резолвятся сразу, очередь RIPE пустеет; до скачивания баз — прежнее поведение.
- Остальные вызовы `lookupRipeCached``resolveFlowIp` (grep: как минимум `traffic-flow-analytics.ts` ~258271).
- `classifyFlowDst`/бренды не трогаем: holder из mmdb (org name) встаёт в существующие `HOLDER_BRANDS`-регулярки как есть.
### 8. Frontend (по next-shadcn-production / ui-guardian: только переиспользование)
- Секция «GeoIP-базы (GeoLite2)» внутри существующей `components/traffic/netflow-settings-panel.tsx`: статус (готово/не скачано, даты сборки Country/ASN, последняя проверка, ошибка), тумблер автообновления, интервал, кнопка «Обновить сейчас» с индикатором. Только уже используемые в панели примитивы (Switch/Button/поля) — никаких новых визуальных паттернов и Card-shell. API-клиент через существующие http-хелперы.
### 9. Хаускипинг, тесты, проверки
- `backend/.gitignore`: `storage/geoip/`.
- Тесты `backend/src/services/traffic-flow-geoip.test.ts` + скрипт `test:geoip` (по образцу `test:traffic-flow`): приоритет фасада (geoip hit → RIPE не зовётся; miss → fallback), negative на приватных IP, фильтрация EU/ZZ через `isIsoCountry`, коллектор с мокнутым fetch (304-skip, битый файл → подмены нет, `.prev` сохранён), dims по стране/ASN с засеянным ридером.
- Проверки после реализации (обязательно по правилам): типы/сборка бэка (`npm run build -w mikrotik-manager-backend`), типы фронта при правке UI (`npx tsc --noEmit`), `npm run test:geoip` и `test:traffic-flow`; предупреждения не игнорировать.
- Коммит: `feat(netflow): <subject по-русски>` — новая пользовательская фича (мгновенные страна/ASN в потоках), по commit-messages-ru.
- README: короткий раздел о GeoIP; примечание, что в Docker `storage/geoip` ephemeral без тома — базы перекачаются после пересоздания контейнера (~17 МБ); при желании смонтировать volume.
## Что это даёт
Страна и ASN появляются у потока мгновенно при ingest (включая IPv6), без ограничения скорости RIPE; dims `country`/`asn` в `flow_daily_dims`, аналитика (карта, топы, monthly) становятся полными сразу. Внешняя зависимость от stat.ripe.net остаётся только как fallback до первой загрузки баз.
+37 -30
View File
@@ -1216,6 +1216,9 @@ export default function NetworkMapPage() {
// ── Interaction ─────────────────────────────────────────────────────────────
const [selected, setSelected] = useState<Server | null>(null)
const [selectedService, setSelectedService] = useState<FlowMapService | null>(null)
const liveSelectedService = selectedService
? (mapServices.find((s) => s.id === selectedService.id) ?? selectedService)
: null
const [highlightedPath, setHighlightedPath] = useState<{ viaId: string; enId: string; serviceId: string } | null>(null)
const [selWanIdx, setSelWanIdx] = useState<number | null>(null)
const [hoveredId, setHoveredId] = useState<string | null>(null)
@@ -2702,16 +2705,16 @@ export default function NetworkMapPage() {
})()}
</div>
</>
) : selectedService ? (
) : liveSelectedService ? (
<>
<div className="flex items-start gap-2 px-4 py-3 border-b">
<div className="mt-0.5">
<ServiceBrandIcon label={selectedService.label} size={22} />
<ServiceBrandIcon label={liveSelectedService.label} size={22} />
</div>
<div className="flex-1 min-w-0">
<p className="font-mono font-semibold text-sm truncate">{selectedService.label}</p>
<p className="font-mono font-semibold text-sm truncate">{liveSelectedService.label}</p>
<p className="text-xs text-muted-foreground mt-0.5">
Конечный сервис · {selectedService.category}
Конечный сервис · {liveSelectedService.category}
</p>
</div>
<button
@@ -2726,15 +2729,15 @@ export default function NetworkMapPage() {
<div className="flex flex-col gap-0">
<div className="flex items-center justify-between py-2 border-b border-border/50">
<span className="text-xs text-muted-foreground">Доля окна</span>
<span className="text-xs font-mono font-medium text-cyan-400">{serviceSharePct(selectedService.share)}</span>
<span className="text-xs font-mono font-medium text-cyan-400">{serviceSharePct(liveSelectedService.share)}</span>
</div>
<div className="flex items-center justify-between py-2 border-b border-border/50">
<span className="text-xs text-muted-foreground">Скорость</span>
<span className="text-xs font-mono font-medium">
{formatNetflowRate({
bytes: selectedService.bytes,
bps: selectedService.bps,
bpsFwd: selectedService.bps,
bytes: liveSelectedService.bytes,
bps: liveSelectedService.bps,
bpsFwd: liveSelectedService.bps,
bpsRev: 0,
})}
</span>
@@ -2742,34 +2745,34 @@ export default function NetworkMapPage() {
</div>
<div>
<p className="text-xs font-semibold text-muted-foreground uppercase tracking-wider mb-2">Выход</p>
<div className="flex flex-col gap-1.5">
{visibleServiceEdges.filter((e) => e.toId === selectedService.id).map((e) => {
<div className="flex flex-col gap-3">
{visibleServiceEdges.filter((e) => e.toId === liveSelectedService.id).map((e) => {
const src = mapServers.find((s) => s.id === e.fromId)
const enPaths = mapServicePaths
.filter((p) => p.serviceId === liveSelectedService.id && p.enId === e.fromId)
.slice()
.sort((a, b) => b.bps - a.bps)
return (
<div key={`${e.fromId}|${e.toId}`} className="flex items-center justify-between text-xs">
<span className="font-mono truncate">{src?.name ?? e.fromId}</span>
<span className="font-mono text-emerald-400 tabular-nums">
{formatNetflowRate({ bytes: e.bytes, bps: e.bps, bpsFwd: e.bpsFwd, bpsRev: e.bpsRev })}
</span>
<div key={`${e.fromId}|${e.toId}`} className="flex flex-col gap-1.5">
<div className="flex items-center justify-between text-xs">
<span className="font-mono truncate">{src?.name ?? e.fromId}</span>
<span className="font-mono text-emerald-400 tabular-nums">
{formatNetflowRate({ bytes: e.bytes, bps: e.bps, bpsFwd: e.bpsFwd, bpsRev: e.bpsRev })}
</span>
</div>
<ServicePathList
paths={enPaths}
servers={mapServers}
services={mapServices}
highlight={highlightedPath}
viaMode="via"
onToggle={togglePathHighlight}
/>
</div>
)
})}
</div>
</div>
<div>
<p className="text-xs font-semibold text-muted-foreground uppercase tracking-wider mb-2">Пути</p>
<ServicePathList
paths={mapServicePaths
.filter((p) => p.serviceId === selectedService.id)
.slice()
.sort((a, b) => b.bps - a.bps)}
servers={mapServers}
services={mapServices}
highlight={highlightedPath}
viaMode="via"
onToggle={togglePathHighlight}
/>
</div>
</div>
</>
) : selected ? (
@@ -3002,7 +3005,11 @@ export default function NetworkMapPage() {
<p className="text-xs font-semibold text-muted-foreground uppercase tracking-wider mb-2">Пути</p>
<ServicePathList
paths={mapServicePaths
.filter((p) => p.viaId === selected.id || p.enId === selected.id)
.filter((p) => (
selected.type === "exit-node"
? p.enId === selected.id
: p.viaId === selected.id
))
.slice()
.sort((a, b) => b.bps - a.bps)}
servers={mapServers}
+237 -142
View File
@@ -4,10 +4,8 @@ import { useCallback, useEffect, useMemo, useState } from "react"
import { useRouter, useSearchParams } from "next/navigation"
import {
ActivityIcon,
CableIcon,
DatabaseIcon,
GaugeIcon,
GlobeIcon,
ServerIcon,
UsersIcon,
} from "lucide-react"
@@ -15,38 +13,45 @@ import { PageHeader } from "@/components/page-header"
import { KpiStatGrid } from "@/components/reui-kit/kpi-stat-grid"
import { DataPageCard } from "@/components/data-page-card"
import { DataPageToolbar } from "@/components/data-page-toolbar"
import { SegmentedControl } from "@/components/form-kit"
import { EmptyState } from "@/components/empty-state"
import { PeriodSelector, rangeForPreset, type DateRangeYmd } from "@/components/statistics/period-selector"
import { StatisticsVolumeChart } from "@/components/statistics/statistics-volume-chart"
import { DimensionSelect, PivotDimSelect } from "@/components/statistics/dimension-select"
import { SliceChips } from "@/components/statistics/slice-chips"
import { BreakdownDashboard } from "@/components/statistics/breakdown-dashboard"
import { StatisticsPivotGrid } from "@/components/statistics/statistics-pivot-grid"
import {
StatisticsBreakdownDataGrid,
type StatisticsSliceKind,
} from "@/components/data-grids/statistics-breakdown-data-grid"
import { Alert, AlertDescription, AlertTitle } from "@/components/reui/alert"
import { Badge } from "@/components/reui/badge"
import type { Filter } from "@/components/reui/filters"
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"
import { STATISTICS_FILTER_FIELDS } from "@/lib/data-filters/statistics-filter-fields"
import {
isStatisticsPivotDim,
isStatisticsSliceKind,
STATISTICS_DIMS,
} from "@/lib/statistics-dims"
import { useDataSource } from "@/lib/data-source"
import { fmtBps, formatBytes } from "@/lib/fmt-rate"
import { getStatistics, type StatisticsDto, type StatisticsQuery } from "@/shared/api/statistics"
import {
getStatistics,
getStatisticsPivot,
STATISTICS_UNBOUND_USER_ID,
type StatisticsDto,
type StatisticsPivotDto,
type StatisticsQuery,
} from "@/shared/api/statistics"
import type { StatisticsBreakdownRow, StatisticsPivotDim } from "@mmapp/contracts/statistics"
/**
* Отчётный куб трафика — KPI + период + график + табы-гриды.
* BI-куб трафика: критерий → остальные разрезы + pivot.
* Preview: https://reui.io/preview/base/dashboard-1 · https://reui.io/preview/base/stats-12
* · https://reui.io/preview/base/data-grid-filtering-2 · https://reui.io/preview/base/chart-23
* · https://reui.io/preview/base/components/c-date-selector-2 · https://reui.io/preview/base/empty-state-12
* · https://reui.io/preview/base/data-grid-filtering-2 · https://reui.io/preview/base/solution-analytics-8
* · https://reui.io/docs/components/base/frame · https://reui.io/docs/components/base/data-grid
*/
const TABS: { id: StatisticsSliceKind; label: string }[] = [
{ id: "users", label: "Пользователи" },
{ id: "servers", label: "Серверы" },
{ id: "interfaces", label: "Интерфейсы" },
{ id: "countries", label: "Страны" },
{ id: "services", label: "Сервисы" },
{ id: "asns", label: "ASN" },
]
const EMPTY: StatisticsDto = {
from: "",
to: "",
@@ -70,6 +75,15 @@ const EMPTY: StatisticsDto = {
asns: [],
}
const EMPTY_PIVOT: StatisticsPivotDto = {
rowDim: "country",
colDim: "service",
metric: "bytes",
columns: [],
rows: [],
otherBytes: 0,
}
interface CubeSlices {
country?: string
service?: string
@@ -88,9 +102,18 @@ function readRange(sp: URLSearchParams): DateRangeYmd {
return rangeForPreset("7d")
}
function readTab(sp: URLSearchParams): StatisticsSliceKind {
const t = sp.get("tab")
return TABS.some((x) => x.id === t) ? (t as StatisticsSliceKind) : "users"
function readDim(sp: URLSearchParams): StatisticsSliceKind {
const t = sp.get("dim") ?? sp.get("tab")
return t && isStatisticsSliceKind(t) ? t : "users"
}
function readView(sp: URLSearchParams): "explore" | "pivot" {
return sp.get("view") === "pivot" ? "pivot" : "explore"
}
function readPivotDim(sp: URLSearchParams, key: string, fallback: StatisticsPivotDim): StatisticsPivotDim {
const v = sp.get(key)
return v && isStatisticsPivotDim(v) ? v : fallback
}
function readSlices(sp: URLSearchParams): CubeSlices {
@@ -140,28 +163,92 @@ function toQuery(range: DateRangeYmd, slices: CubeSlices): StatisticsQuery {
}
}
function selectedIdForTab(tab: StatisticsSliceKind, slices: CubeSlices): string | undefined {
if (tab === "users") return slices.userId
if (tab === "servers") return slices.serverId
if (tab === "countries") return slices.country
if (tab === "services") return slices.service
if (tab === "asns") return slices.asn
if (tab === "interfaces" && slices.serverId && slices.iface) {
function selectedIdForKind(kind: StatisticsSliceKind, slices: CubeSlices): string | undefined {
if (kind === "users") return slices.userId
if (kind === "servers") return slices.serverId
if (kind === "countries") return slices.country
if (kind === "services") return slices.service
if (kind === "asns") return slices.asn
if (kind === "interfaces" && slices.serverId && slices.iface) {
return `${slices.serverId}:${slices.iface}`
}
if (tab === "interfaces") return slices.iface
if (kind === "interfaces") return slices.iface
return undefined
}
function rowsForTab(data: StatisticsDto, tab: StatisticsSliceKind) {
if (tab === "users") return data.users
if (tab === "servers") return data.servers
if (tab === "interfaces") return data.interfaces
if (tab === "countries") return data.countries
if (tab === "services") return data.services
function rowsForKind(data: StatisticsDto, kind: StatisticsSliceKind) {
if (kind === "users") return data.users
if (kind === "servers") return data.servers
if (kind === "interfaces") return data.interfaces
if (kind === "countries") return data.countries
if (kind === "services") return data.services
return data.asns
}
function hasAnySlice(slices: CubeSlices): boolean {
return SLICE_KEYS.some((k) => Boolean(slices[k]))
}
function hiddenKinds(slices: CubeSlices): Set<StatisticsSliceKind> {
const hidden = new Set<StatisticsSliceKind>()
if (slices.userId) hidden.add("users")
if (slices.serverId) hidden.add("servers")
if (slices.iface) hidden.add("interfaces")
if (slices.country) hidden.add("countries")
if (slices.service) hidden.add("services")
if (slices.asn) hidden.add("asns")
return hidden
}
function applyDimValue(slices: CubeSlices, kind: StatisticsSliceKind, rowId: string): CubeSlices {
const next: CubeSlices = { ...slices }
if (kind === "users") {
if (rowId === STATISTICS_UNBOUND_USER_ID) return next
if (next.userId === rowId) delete next.userId
else next.userId = rowId
} else if (kind === "servers") {
if (next.serverId === rowId) delete next.serverId
else next.serverId = rowId
} else if (kind === "countries") {
if (next.country === rowId) delete next.country
else next.country = rowId
} else if (kind === "services") {
if (next.service === rowId) delete next.service
else next.service = rowId
} else if (kind === "asns") {
if (next.asn === rowId) delete next.asn
else next.asn = rowId
} else {
const colon = rowId.indexOf(":")
const sid = colon >= 0 ? rowId.slice(0, colon) : undefined
const iface = colon >= 0 ? rowId.slice(colon + 1) : rowId
if (next.iface === iface && next.serverId === sid) {
delete next.iface
delete next.serverId
} else {
next.iface = iface
if (sid) next.serverId = sid
}
}
return next
}
function applyPivotDim(slices: CubeSlices, dim: StatisticsPivotDim, id: string): CubeSlices {
const kind = STATISTICS_DIMS.find((d) => d.pivot === dim)?.id ?? "users"
return applyDimValue(slices, kind, id)
}
function chipList(slices: CubeSlices): { key: string; label: string }[] {
const chips: { key: string; label: string }[] = []
if (slices.country) chips.push({ key: "country", label: `страна ${slices.country}` })
if (slices.service) chips.push({ key: "service", label: `сервис ${slices.service}` })
if (slices.asn) chips.push({ key: "asn", label: `ASN ${slices.asn}` })
if (slices.serverId) chips.push({ key: "serverId", label: `сервер ${slices.serverId}` })
if (slices.userId) chips.push({ key: "userId", label: `пользователь ${slices.userId}` })
if (slices.iface) chips.push({ key: "iface", label: `iface ${slices.iface}` })
return chips
}
export default function StatisticsPage() {
const router = useRouter()
const searchParams = useSearchParams()
@@ -171,9 +258,13 @@ export default function StatisticsPage() {
const range = useMemo(() => readRange(searchParams), [searchParams])
const slices = useMemo(() => readSlices(searchParams), [searchParams])
const filters = useMemo(() => slicesToFilters(slices), [slices])
const [tab, setTab] = useState<StatisticsSliceKind>(() => readTab(searchParams))
const dim = useMemo(() => readDim(searchParams), [searchParams])
const view = useMemo(() => readView(searchParams), [searchParams])
const pivotRow = useMemo(() => readPivotDim(searchParams, "pivotRow", "country"), [searchParams])
const pivotCol = useMemo(() => readPivotDim(searchParams, "pivotCol", "service"), [searchParams])
const [data, setData] = useState<StatisticsDto>(EMPTY)
const [pivot, setPivot] = useState<StatisticsPivotDto>(EMPTY_PIVOT)
const [loading, setLoading] = useState(false)
const [error, setError] = useState<string | null>(null)
@@ -218,11 +309,22 @@ export default function StatisticsPage() {
setLoading(true)
setError(null)
try {
const dto = await getStatistics(backendUrl, toQuery(range, slices))
const query = toQuery(range, slices)
const dto = await getStatistics(backendUrl, query)
if (!cancelled) setData(dto)
if (view === "pivot" && pivotRow !== pivotCol) {
const matrix = await getStatisticsPivot(backendUrl, {
...query,
row: pivotRow,
col: pivotCol,
metric: "bytes",
})
if (!cancelled) setPivot(matrix)
}
} catch (e: unknown) {
if (!cancelled) {
setData(EMPTY)
setPivot(EMPTY_PIVOT)
setError(e instanceof Error ? e.message : "Не удалось загрузить статистику")
}
} finally {
@@ -232,46 +334,40 @@ export default function StatisticsPage() {
return () => {
cancelled = true
}
}, [backendUrl, isLive, prefsHydrated, range, slices])
}, [backendUrl, isLive, prefsHydrated, range, slices, view, pivotRow, pivotCol])
const rows = rowsForTab(isLive ? data : EMPTY, tab)
const selectedId = selectedIdForTab(tab, slices)
const view = isLive ? data : EMPTY
const viewData = isLive ? data : EMPTY
const sliced = hasAnySlice(slices)
const emptyCube = !isLive || (!loading && viewData.kpis.bytes === 0)
function handleRowClick(kind: StatisticsSliceKind, row: { id: string }) {
const next: CubeSlices = { ...slices }
if (kind === "users") {
if (next.userId === row.id) delete next.userId
else next.userId = row.id
} else if (kind === "servers") {
if (next.serverId === row.id) delete next.serverId
else next.serverId = row.id
} else if (kind === "countries") {
if (next.country === row.id) delete next.country
else next.country = row.id
} else if (kind === "services") {
if (next.service === row.id) delete next.service
else next.service = row.id
} else if (kind === "asns") {
if (next.asn === row.id) delete next.asn
else next.asn = row.id
} else {
const colon = row.id.indexOf(":")
const sid = colon >= 0 ? row.id.slice(0, colon) : undefined
const iface = colon >= 0 ? row.id.slice(colon + 1) : row.id
if (next.iface === iface && next.serverId === sid) {
delete next.iface
delete next.serverId
} else {
next.iface = iface
if (sid) next.serverId = sid
}
}
setSlices(next)
function handleRowClick(kind: StatisticsSliceKind, row: StatisticsBreakdownRow) {
if (kind === "users" && row.id === STATISTICS_UNBOUND_USER_ID) return
setSlices(applyDimValue(slices, kind, row.id))
}
const kpis = view.kpis
const emptyCube = !isLive || (!loading && kpis.bytes === 0)
function handlePivotCell(rowId: string, colId: string) {
if (rowId === "__other__" || colId === "__other__") return
let next = applyPivotDim(slices, pivotRow, rowId)
next = applyPivotDim(next, pivotCol, colId)
replaceParams({
country: next.country,
service: next.service,
asn: next.asn,
serverId: next.serverId,
userId: next.userId,
iface: next.iface,
view: "explore",
})
}
const kpis = viewData.kpis
const chips = chipList(slices)
const countLabel =
view === "pivot"
? `${pivot.rows.length} × ${pivot.columns.length}`
: sliced
? `${STATISTICS_DIMS.filter((d) => !hiddenKinds(slices).has(d.id)).length} разрезов`
: `${rowsForKind(viewData, dim).length} строк`
return (
<div className="flex h-full flex-col">
@@ -343,83 +439,82 @@ export default function StatisticsPage() {
]}
/>
<StatisticsVolumeChart series={view.series} grain={view.grain} />
<StatisticsVolumeChart series={viewData.series} grain={viewData.grain} />
<DataPageCard>
<DataPageToolbar
leading={
<div className="flex flex-wrap items-center gap-3">
<SegmentedControl
value={view}
onChange={(next) => replaceParams({ view: next === "pivot" ? "pivot" : "explore" })}
options={[
{ value: "explore", label: "Разрез" },
{ value: "pivot", label: "Сводка" },
]}
/>
{view === "explore" && !sliced ? (
<DimensionSelect
label="Критерий"
value={dim}
onChange={(next) => replaceParams({ dim: next })}
/>
) : null}
{view === "pivot" ? (
<>
<PivotDimSelect
label="Строки"
value={pivotRow}
exclude={pivotCol}
onChange={(next) => replaceParams({ pivotRow: next })}
/>
<PivotDimSelect
label="Колонки"
value={pivotCol}
exclude={pivotRow}
onChange={(next) => replaceParams({ pivotCol: next })}
/>
</>
) : null}
</div>
}
filters={filters}
onFiltersChange={(next) => setSlices(filtersToSlices(next))}
filterFields={STATISTICS_FILTER_FIELDS}
countLabel={`${rows.length} строк`}
countLabel={countLabel}
/>
{SLICE_KEYS.some((k) => slices[k]) ? (
<div className="flex flex-wrap items-center gap-1.5 border-b px-5 py-2">
{slices.country ? (
<Badge variant="outline" size="sm">страна {slices.country}</Badge>
) : null}
{slices.service ? (
<Badge variant="outline" size="sm">сервис {slices.service}</Badge>
) : null}
{slices.asn ? (
<Badge variant="outline" size="sm">ASN {slices.asn}</Badge>
) : null}
{slices.serverId ? (
<Badge variant="outline" size="sm">сервер {slices.serverId}</Badge>
) : null}
{slices.userId ? (
<Badge variant="outline" size="sm">пользователь {slices.userId}</Badge>
) : null}
{slices.iface ? (
<Badge variant="outline" size="sm">iface {slices.iface}</Badge>
) : null}
</div>
) : null}
<Tabs
value={tab}
onValueChange={(v) => {
const next = String(v) as StatisticsSliceKind
setTab(next)
replaceParams({ tab: next })
}}
className="gap-0"
>
<div className="px-5 pt-2">
<TabsList variant="line" className="w-fit">
<TabsTrigger value="users">
<UsersIcon /> Пользователи
</TabsTrigger>
<TabsTrigger value="servers">
<ServerIcon /> Серверы
</TabsTrigger>
<TabsTrigger value="interfaces">
<CableIcon /> Интерфейсы
</TabsTrigger>
<TabsTrigger value="countries">
<GlobeIcon /> Страны
</TabsTrigger>
<TabsTrigger value="services">Сервисы</TabsTrigger>
<TabsTrigger value="asns">ASN</TabsTrigger>
</TabsList>
</div>
{TABS.map((t) => (
<TabsContent key={t.id} value={t.id}>
{emptyCube ? (
<EmptyState
title="Нет данных куба"
description="За выбранный период нет IPFIX-фактов. Куб заполняется с момента деплоя, без бэкфилла за год."
/>
) : (
<StatisticsBreakdownDataGrid
rows={rowsForTab(view, t.id)}
kind={t.id}
selectedId={t.id === tab ? selectedId : undefined}
onRowClick={(row) => handleRowClick(t.id, row)}
isLoading={loading}
/>
)}
</TabsContent>
))}
</Tabs>
<SliceChips
chips={chips}
onRemove={(key) => {
const next = { ...slices }
delete next[key as keyof CubeSlices]
setSlices(next)
}}
/>
{emptyCube ? (
<EmptyState
title="Нет данных куба"
description="За выбранный период нет IPFIX-фактов. Куб заполняется с момента деплоя, без бэкфилла за год."
/>
) : view === "pivot" ? (
<StatisticsPivotGrid data={isLive ? pivot : EMPTY_PIVOT} onCellClick={handlePivotCell} isLoading={loading} />
) : sliced ? (
<BreakdownDashboard
data={viewData}
hidden={hiddenKinds(slices)}
selectedIdFor={(kind) => selectedIdForKind(kind, slices)}
onRowClick={handleRowClick}
isLoading={loading}
/>
) : (
<StatisticsBreakdownDataGrid
rows={rowsForKind(viewData, dim)}
kind={dim}
selectedId={selectedIdForKind(dim, slices)}
onRowClick={(row) => handleRowClick(dim, row)}
isLoading={loading}
/>
)}
</DataPageCard>
</div>
</div>
+1 -1
View File
@@ -15,7 +15,7 @@
"test:auth": "tsx src/lib/permissions.test.ts && tsx src/plugins/auth.smoke.test.ts",
"test:wireguard": "npx tsx src/services/wireguard-config.test.ts",
"test:traffic-rate": "tsx src/services/traffic-rate.test.ts",
"test:traffic-flow": "tsx src/services/traffic-flow-parse.test.ts && tsx src/services/traffic-flow-map-exporter.test.ts && tsx src/services/traffic-flow-ifaces.test.ts && tsx src/services/traffic-flow-dedup.test.ts && tsx src/services/traffic-flow-planes.test.ts && tsx src/services/traffic-flow-ip.test.ts && tsx src/services/traffic-flow-classify.test.ts && tsx src/services/traffic-flow-ripe.test.ts && tsx src/services/traffic-flow-brands.test.ts && tsx src/services/traffic-flow-ingest.test.ts && tsx src/services/traffic-flow-analytics.test.ts && tsx src/services/traffic-flow-map-hops.test.ts && tsx src/services/traffic-flow-purge.test.ts && tsx src/services/traffic-flow-geoip.test.ts && tsx src/services/traffic-flow-facts.test.ts && tsx src/services/statistics-aggregate.test.ts",
"test:traffic-flow": "tsx src/services/traffic-flow-parse.test.ts && tsx src/services/traffic-flow-map-exporter.test.ts && tsx src/services/traffic-flow-ifaces.test.ts && tsx src/services/traffic-flow-ifindex.test.ts && tsx src/services/traffic-flow-dedup.test.ts && tsx src/services/traffic-flow-planes.test.ts && tsx src/services/traffic-flow-ip.test.ts && tsx src/services/traffic-flow-classify.test.ts && tsx src/services/traffic-flow-ripe.test.ts && tsx src/services/traffic-flow-brands.test.ts && tsx src/services/traffic-flow-ingest.test.ts && tsx src/services/traffic-flow-analytics.test.ts && tsx src/services/traffic-flow-map-hops.test.ts && tsx src/services/traffic-flow-purge.test.ts && tsx src/services/traffic-flow-geoip.test.ts && tsx src/services/traffic-flow-facts.test.ts && tsx src/services/statistics-aggregate.test.ts",
"test:users": "tsx src/modules/users/iface-type.test.ts && tsx src/modules/users/bindings.test.ts",
"test:pg": "tsx src/db/sql-bind.test.ts && tsx src/db/sqlite-json.test.ts && tsx src/db/traffic-flags.test.ts && tsx src/db/pg-schema.test.ts",
"test:backups": "tsx src/services/s3-backup-client.test.ts",
+13 -2
View File
@@ -1,6 +1,6 @@
import type { FastifyPluginAsyncZod } from "@fastify/type-provider-zod"
import { statisticsQuerySchema } from "@mmapp/contracts/statistics"
import { getStatistics } from "../services/statistics-aggregate.js"
import { statisticsPivotQuerySchema, statisticsQuerySchema } from "@mmapp/contracts/statistics"
import { getStatistics, getStatisticsPivot, pivotDimsConflict } from "../services/statistics-aggregate.js"
const statisticsRoutes: FastifyPluginAsyncZod = async (app) => {
app.get("/statistics", async (req, reply) => {
@@ -10,6 +10,17 @@ const statisticsRoutes: FastifyPluginAsyncZod = async (app) => {
}
return reply.send(await getStatistics(parsed.data))
})
app.get("/statistics/pivot", async (req, reply) => {
const parsed = statisticsPivotQuerySchema.safeParse(req.query ?? {})
if (!parsed.success) {
return reply.status(400).send({ error: "Некорректный период или измерения", details: parsed.error.flatten() })
}
if (pivotDimsConflict(parsed.data.row, parsed.data.col)) {
return reply.status(400).send({ error: "Строки и колонки должны отличаться" })
}
return reply.send(await getStatisticsPivot(parsed.data))
})
}
export default statisticsRoutes
@@ -1,9 +1,12 @@
import assert from "node:assert/strict"
import { getStatistics, parseStatisticsPeriod } from "./statistics-aggregate.js"
import { getStatistics, getStatisticsPivot, parseStatisticsPeriod, pivotDimsConflict } from "./statistics-aggregate.js"
import { rememberServerIfaces, resetIfaceCacheForTests } from "./traffic-flow-ifindex.js"
import { setRefreshIfacesForTests } from "./traffic-flow-ifaces.js"
import { withPgOrSkip } from "../test/pg.js"
import { dbQuery } from "../db/index.js"
import { ensurePartitionFor } from "../db/partitions.js"
import { pool } from "../db/index.js"
import { STATISTICS_UNBOUND_USER_ID } from "@mmapp/contracts/statistics"
{
const sameDay = parseStatisticsPeriod("2026-09-10", "2026-09-10")
@@ -16,6 +19,8 @@ import { pool } from "../db/index.js"
assert.equal(month.grain, "day")
assert.equal(month.toDayExclusive, "2026-09-01")
assert.equal(parseStatisticsPeriod("2026-09-10", "2026-09-09"), null)
assert.equal(pivotDimsConflict("country", "country"), true)
assert.equal(pivotDimsConflict("country", "service"), false)
}
if (!(await withPgOrSkip())) {
@@ -43,23 +48,37 @@ await dbQuery(`
`)
await dbQuery(`
INSERT INTO user_interface_bindings (id, user_id, server_id, interface_name, interface_type)
VALUES ('bind-stats-1', 'u-stats-1', $1, 'ether1', 'ether')
VALUES ('bind-stats-1', 'u-stats-1', $1, 'gre-client', 'gre')
`, [serverId])
resetIfaceCacheForTests()
rememberServerIfaces(serverId, [{ name: "gre-client", ifindex: "2" }])
setRefreshIfacesForTests(async () => {})
await dbQuery(`
INSERT INTO flow_daily_facts (server_id, day, iface, country, service, asn, bytes, packets)
VALUES
($1, '2026-09-10', 'ether1', 'US', 'https', 15169, 800, 10),
($1, '2026-09-10', 'ether1', 'DE', 'dns', 15133, 200, 4)
($1, '2026-09-10', '2', 'US', 'https', 15169, 800, 10),
($1, '2026-09-10', '2', 'DE', 'dns', 15133, 200, 4),
($1, '2026-09-10', 'wan1', 'NL', 'other', 0, 70, 1)
`, [serverId])
try {
const all = await getStatistics({ from: "2026-09-01", to: "2026-09-30" })
assert.equal(all.grain, "day")
assert.equal(all.kpis.bytes, 1000)
assert.equal(all.kpis.bytes, 1070)
assert.equal(all.kpis.users, 1)
assert.ok(all.countries.some((r) => r.id === "US"))
assert.ok(all.users.some((r) => r.id === "u-stats-1"))
const unbound = all.users.find((r) => r.id === STATISTICS_UNBOUND_USER_ID)
assert.ok(unbound)
assert.equal(unbound.bytes, 70)
assert.ok(all.servers.some((r) => r.id === String(serverId)))
const greIface = all.interfaces.find((r) => r.label.includes("gre-client"))
assert.ok(greIface)
assert.equal(greIface.bytes, 1000)
assert.equal(greIface.id, `${serverId}:gre-client`)
assert.ok(!all.interfaces.some((r) => /· (?:#)?\d+$/.test(r.label)))
const sliced = await getStatistics({
from: "2026-09-01",
@@ -73,9 +92,30 @@ try {
assert.equal(sliced.countries[0]?.id, "US")
assert.ok(sliced.users.some((r) => r.id === "u-stats-1"))
const byUser = await getStatistics({
from: "2026-09-01",
to: "2026-09-30",
userId: "u-stats-1",
})
assert.equal(byUser.kpis.bytes, 1000)
const pivot = await getStatisticsPivot({
from: "2026-09-01",
to: "2026-09-30",
row: "country",
col: "service",
metric: "bytes",
})
const us = pivot.rows.find((r) => r.id === "US")
const de = pivot.rows.find((r) => r.id === "DE")
assert.ok(us)
assert.ok(de)
assert.equal(us.cells.https, 800)
assert.equal(de.cells.dns, 200)
await dbQuery(`
INSERT INTO flow_hour_facts (server_id, bucket_at, iface, country, service, asn, bytes, packets)
VALUES ($1, '2026-09-10T10:00:00Z', 'ether1', 'US', 'https', 15169, 40, 2)
VALUES ($1, '2026-09-10T10:00:00Z', '2', 'US', 'https', 15169, 40, 2)
`, [serverId])
const hourly = await getStatistics({
from: "2026-09-10T00:00:00.000Z",
@@ -83,7 +123,10 @@ try {
})
assert.equal(hourly.grain, "hour")
assert.equal(hourly.kpis.bytes, 40)
assert.ok(hourly.users.some((r) => r.id === "u-stats-1"))
} finally {
setRefreshIfacesForTests(null)
resetIfaceCacheForTests()
await dbQuery(`DELETE FROM flow_daily_facts WHERE server_id = $1`, [serverId])
await dbQuery(`DELETE FROM flow_hour_facts WHERE server_id = $1`, [serverId])
await dbQuery(`DELETE FROM servers WHERE id = $1`, [serverId])
+413 -36
View File
@@ -1,14 +1,28 @@
import { eq } from "drizzle-orm"
import { db, dbAll } from "../db/index.js"
import { appUsers, flowAsnMeta, servers, userInterfaceBindings } from "../db/schema.js"
import type {
StatisticsBreakdownRow,
StatisticsDto,
StatisticsQuery,
import {
STATISTICS_UNBOUND_USER_ID,
type StatisticsBreakdownRow,
type StatisticsDto,
type StatisticsPivotDim,
type StatisticsPivotDto,
type StatisticsPivotQuery,
type StatisticsQuery,
} from "@mmapp/contracts/statistics"
import {
collapseServerIfaceRows,
displayFactIface,
expandBindingIfaces,
factIfaceAliases,
} from "./traffic-flow-ifindex.js"
import { refreshServerIfaces } from "./traffic-flow-ifaces.js"
const TOP_N = 200
const HOUR_WINDOW_MS = 48 * 3600_000
const PIVOT_ROW_CAP = 50
const PIVOT_COL_CAP = 15
const PIVOT_OTHER_ID = "__other__"
export interface ParsedPeriod {
fromIso: string
@@ -75,6 +89,36 @@ interface FilterCtx {
service?: string
asn?: number
userIfaces: Array<{ serverId: number; iface: string }> | null
unboundOnly: boolean
boundIfaces: Array<{ serverId: number; iface: string }>
}
function ifaceFilterAliases(iface: string, serverId?: number): string[] {
return factIfaceAliases(iface.trim(), serverId)
}
function looksLikeIfIndex(iface: string): boolean {
const raw = iface.trim()
return /^\d+$/.test(raw) || /^#\d+$/.test(raw)
}
async function warmIfaceCache(ids: Iterable<number>): Promise<void> {
const uniq = [...new Set(ids)].filter((id) => Number.isFinite(id) && id > 0)
if (!uniq.length) return
await Promise.all(uniq.map((id) => refreshServerIfaces(id)))
}
async function warmBindingIfaceCache(): Promise<void> {
const rows = await db.select({ serverId: userInterfaceBindings.serverId }).from(userInterfaceBindings)
await warmIfaceCache(rows.map((r) => r.serverId))
}
function canonicalIfaceDimId(id: string): string {
const colon = id.indexOf(":")
if (colon < 0) return id
const sid = Number(id.slice(0, colon))
if (!Number.isFinite(sid)) return id
return `${sid}:${displayFactIface(sid, id.slice(colon + 1))}`
}
function factWhere(alias: string, grain: "hour" | "day", ctx: FilterCtx): { sql: string; params: unknown[] } {
@@ -92,8 +136,14 @@ function factWhere(alias: string, grain: "hour" | "day", ctx: FilterCtx): { sql:
params.push(ctx.serverId)
}
if (ctx.iface) {
parts.push(`${alias}.iface = ?`)
params.push(ctx.iface)
const aliases = ifaceFilterAliases(ctx.iface, ctx.serverId)
if (aliases.length <= 1) {
parts.push(`${alias}.iface = ?`)
params.push(aliases[0] ?? ctx.iface)
} else {
parts.push(`${alias}.iface IN (${aliases.map(() => "?").join(", ")})`)
params.push(...aliases)
}
}
if (ctx.country) {
parts.push(`${alias}.country = ?`)
@@ -118,11 +168,20 @@ function factWhere(alias: string, grain: "hour" | "day", ctx: FilterCtx): { sql:
}
}
}
if (ctx.unboundOnly) {
if (ctx.boundIfaces.length === 0) {
/* весь трафик без привязок */
} else {
const tuples = ctx.boundIfaces.map(() => "(?, ?)").join(", ")
parts.push(`(${alias}.server_id, ${alias}.iface) NOT IN (${tuples})`)
for (const u of ctx.boundIfaces) {
params.push(u.serverId, u.iface)
}
}
}
return { sql: parts.join(" AND "), params }
}
type FilterCtxFull = FilterCtx
function emptyDto(period: ParsedPeriod): StatisticsDto {
return {
from: period.fromIso,
@@ -167,10 +226,71 @@ function toBreakdown(
}))
}
interface UserBindTuple {
userId: string
serverId: number
iface: string
}
async function loadBindUserTuples(): Promise<UserBindTuple[]> {
const binds = await db.select().from(userInterfaceBindings)
const seen = new Set<string>()
const out: UserBindTuple[] = []
for (const b of binds) {
for (const iface of factIfaceAliases(b.interfaceName, b.serverId)) {
const k = `${b.userId}\0${b.serverId}\0${iface}`
if (seen.has(k)) continue
seen.add(k)
out.push({ userId: b.userId, serverId: b.serverId, iface })
}
}
return out
}
function uniqueBoundIfaces(tuples: UserBindTuple[]): Array<{ serverId: number; iface: string }> {
const seen = new Set<string>()
const out: Array<{ serverId: number; iface: string }> = []
for (const t of tuples) {
const k = `${t.serverId}\0${t.iface}`
if (seen.has(k)) continue
seen.add(k)
out.push({ serverId: t.serverId, iface: t.iface })
}
return out
}
async function resolveUserIfaces(userId?: string): Promise<Array<{ serverId: number; iface: string }> | null> {
if (!userId) return null
if (!userId || userId === STATISTICS_UNBOUND_USER_ID) return null
const binds = await db.select().from(userInterfaceBindings).where(eq(userInterfaceBindings.userId, userId))
return binds.map((b) => ({ serverId: b.serverId, iface: b.interfaceName }))
return expandBindingIfaces(binds.map((b) => ({ serverId: b.serverId, iface: b.interfaceName })))
}
function userBindJoinSql(tuples: UserBindTuple[]): { sql: string; params: unknown[] } {
const values = tuples.map(() => "(?::text, ?::int, ?::text)").join(", ")
const params = tuples.flatMap((t) => [t.userId, t.serverId, t.iface])
return {
sql: `JOIN (VALUES ${values}) AS b(user_id, server_id, iface) ON b.server_id = f.server_id AND b.iface = f.iface`,
params,
}
}
async function buildFilterCtx(query: StatisticsQuery, period: ParsedPeriod): Promise<FilterCtx | null> {
const bindTuples = await loadBindUserTuples()
const boundIfaces = uniqueBoundIfaces(bindTuples)
const unboundOnly = query.userId === STATISTICS_UNBOUND_USER_ID
const userIfaces = unboundOnly ? null : await resolveUserIfaces(query.userId)
if (userIfaces && userIfaces.length === 0) return null
return {
...period,
serverId: query.serverId,
iface: query.iface,
country: query.country,
service: query.service,
asn: query.asn,
userIfaces,
unboundOnly,
boundIfaces,
}
}
export async function getStatistics(query: StatisticsQuery): Promise<StatisticsDto> {
@@ -184,28 +304,21 @@ export async function getStatistics(query: StatisticsQuery): Promise<StatisticsD
windowSec: 1,
})
const userIfaces = await resolveUserIfaces(query.userId)
const ctx: FilterCtxFull = {
...period,
serverId: query.serverId,
iface: query.iface,
country: query.country,
service: query.service,
asn: query.asn,
userIfaces,
}
if (userIfaces && userIfaces.length === 0) return emptyDto(period)
await warmBindingIfaceCache()
if (query.serverId) await warmIfaceCache([query.serverId])
const bindTuples = await loadBindUserTuples()
const ctx = await buildFilterCtx(query, period)
if (!ctx) return emptyDto(period)
const table = period.grain === "hour" ? "flow_hour_facts" : "flow_daily_facts"
const timeCol = period.grain === "hour" ? "bucket_at" : "day"
const where = factWhere("f", period.grain, ctx)
const totals = await dbAll<{ bytes: number; packets: number; servers: number; ifaces: number }>(`
const totals = await dbAll<{ bytes: number; packets: number; servers: number }>(`
SELECT
COALESCE(SUM(f.bytes), 0) AS bytes,
COALESCE(SUM(f.packets), 0) AS packets,
COUNT(DISTINCT f.server_id)::int AS servers,
COUNT(DISTINCT (f.server_id::text || ':' || f.iface))::int AS ifaces
COUNT(DISTINCT f.server_id)::int AS servers
FROM ${table} f
WHERE ${where.sql}
`, where.params)
@@ -213,7 +326,6 @@ export async function getStatistics(query: StatisticsQuery): Promise<StatisticsD
const bytes = Number(totals[0]?.bytes) || 0
const packets = Number(totals[0]?.packets) || 0
const serverCount = Number(totals[0]?.servers) || 0
const ifaceCount = Number(totals[0]?.ifaces) || 0
const seriesRows = await dbAll<{ t: string; bytes: number }>(`
SELECT ${timeCol}::text AS t, SUM(f.bytes) AS bytes
@@ -251,21 +363,27 @@ export async function getStatistics(query: StatisticsQuery): Promise<StatisticsD
GROUP BY f.server_id
`, where.params)
const ifaceRows = await dbAll<{ serverId: number; iface: string; bytes: number; packets: number }>(`
const ifaceRowsRaw = await dbAll<{ serverId: number; iface: string; bytes: number; packets: number }>(`
SELECT f.server_id AS "serverId", f.iface AS iface, SUM(f.bytes) AS bytes, SUM(f.packets) AS packets
FROM ${table} f
WHERE ${where.sql}
GROUP BY f.server_id, f.iface
`, where.params)
await warmIfaceCache(ifaceRowsRaw.filter((r) => looksLikeIfIndex(r.iface)).map((r) => r.serverId))
const ifaceRows = collapseServerIfaceRows(ifaceRowsRaw)
const ifaceCount = ifaceRows.length
const userRows = await dbAll<{ id: string; bytes: number; packets: number }>(`
SELECT b.user_id AS id, SUM(f.bytes) AS bytes, SUM(f.packets) AS packets
FROM ${table} f
JOIN user_interface_bindings b
ON b.server_id = f.server_id AND b.interface_name = f.iface
WHERE ${where.sql}
GROUP BY b.user_id
`, where.params)
let userRows: Array<{ id: string; bytes: number; packets: number }> = []
if (bindTuples.length && !ctx.unboundOnly) {
const join = userBindJoinSql(bindTuples)
userRows = await dbAll<{ id: string; bytes: number; packets: number }>(`
SELECT b.user_id AS id, SUM(f.bytes) AS bytes, SUM(f.packets) AS packets
FROM ${table} f
${join.sql}
WHERE ${where.sql}
GROUP BY b.user_id
`, [...join.params, ...where.params])
}
const serverNames = new Map<number, string>()
const allServers = await db.select({ id: servers.id, name: servers.name, host: servers.host }).from(servers)
@@ -333,7 +451,7 @@ export async function getStatistics(query: StatisticsQuery): Promise<StatisticsD
bytes,
period.windowSec,
)
const users = toBreakdown(
const matchedUsers = toBreakdown(
userRows.map((r) => ({
id: r.id,
label: userNames.get(r.id) || r.id,
@@ -344,6 +462,40 @@ export async function getStatistics(query: StatisticsQuery): Promise<StatisticsD
period.windowSec,
)
const users = [...matchedUsers]
if (!ctx.unboundOnly && !ctx.userIfaces) {
let unboundBytes = 0
let unboundPackets = 0
if (ctx.boundIfaces.length === 0) {
unboundBytes = bytes
unboundPackets = packets
} else {
const tuples = ctx.boundIfaces.map(() => "(?, ?)").join(", ")
const unboundParams = [...where.params]
for (const u of ctx.boundIfaces) unboundParams.push(u.serverId, u.iface)
const unboundRows = await dbAll<{ bytes: number; packets: number }>(`
SELECT COALESCE(SUM(f.bytes), 0) AS bytes, COALESCE(SUM(f.packets), 0) AS packets
FROM ${table} f
WHERE ${where.sql}
AND (f.server_id, f.iface) NOT IN (${tuples})
`, unboundParams)
unboundBytes = Number(unboundRows[0]?.bytes) || 0
unboundPackets = Number(unboundRows[0]?.packets) || 0
}
if (unboundBytes > 0) {
const denom = bytes || 1
users.push({
id: STATISTICS_UNBOUND_USER_ID,
label: "Без привязки",
bytes: unboundBytes,
packets: unboundPackets,
bps: (unboundBytes * 8) / period.windowSec,
percent: (unboundBytes / denom) * 100,
})
users.sort((a, b) => b.bytes - a.bytes)
}
}
return {
from: period.fromIso,
to: period.toIso,
@@ -352,7 +504,7 @@ export async function getStatistics(query: StatisticsQuery): Promise<StatisticsD
bytes,
packets,
avgBps: (bytes * 8) / period.windowSec,
users: users.length,
users: matchedUsers.length,
servers: serverCount,
ifaces: ifaceCount,
topCountry: countries[0]?.label || "—",
@@ -367,3 +519,228 @@ export async function getStatistics(query: StatisticsQuery): Promise<StatisticsD
asns,
}
}
function dimSql(dim: StatisticsPivotDim, factAlias: string, bindAlias: string): string {
if (dim === "country") return `${factAlias}.country`
if (dim === "service") return `${factAlias}.service`
if (dim === "asn") return `${factAlias}.asn::text`
if (dim === "server") return `${factAlias}.server_id::text`
if (dim === "iface") return `(${factAlias}.server_id::text || ':' || ${factAlias}.iface)`
return `${bindAlias}.user_id`
}
function emptyPivot(query: StatisticsPivotQuery): StatisticsPivotDto {
return {
rowDim: query.row,
colDim: query.col,
metric: query.metric,
columns: [],
rows: [],
otherBytes: 0,
}
}
export function pivotDimsConflict(row: StatisticsPivotDim, col: StatisticsPivotDim): boolean {
return row === col
}
export async function getStatisticsPivot(query: StatisticsPivotQuery): Promise<StatisticsPivotDto> {
if (pivotDimsConflict(query.row, query.col)) return emptyPivot(query)
const period = parseStatisticsPeriod(query.from, query.to)
if (!period) return emptyPivot(query)
await warmBindingIfaceCache()
if (query.serverId) await warmIfaceCache([query.serverId])
const bindTuples = await loadBindUserTuples()
const ctx = await buildFilterCtx(query, period)
if (!ctx) return emptyPivot(query)
const needsUser = query.row === "user" || query.col === "user"
if (needsUser && bindTuples.length === 0) return emptyPivot(query)
const table = period.grain === "hour" ? "flow_hour_facts" : "flow_daily_facts"
const where = factWhere("f", period.grain, ctx)
const rowExpr = dimSql(query.row, "f", "b")
const colExpr = dimSql(query.col, "f", "b")
const join = needsUser ? userBindJoinSql(bindTuples) : { sql: "", params: [] as unknown[] }
const raw = await dbAll<{ row_id: string; col_id: string; bytes: number; packets: number }>(`
SELECT ${rowExpr} AS row_id, ${colExpr} AS col_id,
SUM(f.bytes) AS bytes, SUM(f.packets) AS packets
FROM ${table} f
${join.sql}
WHERE ${where.sql}
GROUP BY 1, 2
`, [...join.params, ...where.params])
if (query.row === "iface" || query.col === "iface") {
const ifaceServerIds: number[] = []
for (const r of raw) {
for (const dim of [query.row, query.col] as const) {
if (dim !== "iface") continue
const id = dim === query.row ? String(r.row_id ?? "") : String(r.col_id ?? "")
const colon = id.indexOf(":")
if (colon < 0) continue
const sid = Number(id.slice(0, colon))
if (looksLikeIfIndex(id.slice(colon + 1)) && Number.isFinite(sid)) ifaceServerIds.push(sid)
}
}
await warmIfaceCache(ifaceServerIds)
for (const r of raw) {
if (query.row === "iface") r.row_id = canonicalIfaceDimId(String(r.row_id ?? ""))
if (query.col === "iface") r.col_id = canonicalIfaceDimId(String(r.col_id ?? ""))
}
}
const metric = query.metric
type Acc = { bytes: number; packets: number }
const cell = new Map<string, Map<string, Acc>>()
const colTotals = new Map<string, number>()
for (const r of raw) {
const rid = String(r.row_id ?? "")
const cid = String(r.col_id ?? "")
const acc: Acc = { bytes: Number(r.bytes) || 0, packets: Number(r.packets) || 0 }
const val = metric === "packets" ? acc.packets : acc.bytes
let rowMap = cell.get(rid)
if (!rowMap) {
rowMap = new Map()
cell.set(rid, rowMap)
}
const prev = rowMap.get(cid)
if (prev) {
prev.bytes += acc.bytes
prev.packets += acc.packets
} else {
rowMap.set(cid, acc)
}
colTotals.set(cid, (colTotals.get(cid) ?? 0) + val)
}
const topCols = [...colTotals.entries()]
.sort((a, b) => b[1] - a[1])
.slice(0, PIVOT_COL_CAP)
.map(([id]) => id)
const topColSet = new Set(topCols)
const folded = new Map<string, Map<string, number>>()
const foldedColTotals = new Map<string, number>()
let otherBytes = 0
for (const [rid, cols] of cell) {
const rowMap = new Map<string, number>()
for (const [cid, acc] of cols) {
const val = metric === "packets" ? acc.packets : acc.bytes
const dest = topColSet.has(cid) ? cid : PIVOT_OTHER_ID
if (dest === PIVOT_OTHER_ID) otherBytes += val
rowMap.set(dest, (rowMap.get(dest) ?? 0) + val)
foldedColTotals.set(dest, (foldedColTotals.get(dest) ?? 0) + val)
}
folded.set(rid, rowMap)
}
const rowTotals = new Map<string, number>()
for (const [rid, cols] of folded) {
let t = 0
for (const v of cols.values()) t += v
rowTotals.set(rid, t)
}
const topRows = [...rowTotals.entries()]
.sort((a, b) => b[1] - a[1])
.slice(0, PIVOT_ROW_CAP)
.map(([id]) => id)
const topRowSet = new Set(topRows)
const finalRows = new Map<string, Map<string, number>>()
const finalRowTotals = new Map<string, number>()
for (const [rid, cols] of folded) {
const dest = topRowSet.has(rid) ? rid : PIVOT_OTHER_ID
if (dest === PIVOT_OTHER_ID) {
for (const [cid, v] of cols) {
if (cid !== PIVOT_OTHER_ID) otherBytes += v
}
}
let rowMap = finalRows.get(dest)
if (!rowMap) {
rowMap = new Map()
finalRows.set(dest, rowMap)
}
for (const [cid, v] of cols) {
rowMap.set(cid, (rowMap.get(cid) ?? 0) + v)
}
}
for (const [rid, cols] of finalRows) {
let t = 0
for (const v of cols.values()) t += v
finalRowTotals.set(rid, t)
}
const colIds = [...topCols]
if (foldedColTotals.has(PIVOT_OTHER_ID)) colIds.push(PIVOT_OTHER_ID)
const rowIds = [...topRows]
if (finalRows.has(PIVOT_OTHER_ID) && !topRowSet.has(PIVOT_OTHER_ID)) rowIds.push(PIVOT_OTHER_ID)
const labels = await loadPivotLabels(query.row, query.col, rowIds, colIds)
return {
rowDim: query.row,
colDim: query.col,
metric,
columns: colIds.map((id) => ({
id,
label: labels.col.get(id) ?? (id === PIVOT_OTHER_ID ? "Прочие" : id),
total: foldedColTotals.get(id) ?? 0,
})),
rows: rowIds.map((id) => {
const cols = finalRows.get(id) ?? new Map()
const cells: Record<string, number> = {}
for (const cid of colIds) cells[cid] = cols.get(cid) ?? 0
return {
id,
label: labels.row.get(id) ?? (id === PIVOT_OTHER_ID ? "Прочие" : id),
total: finalRowTotals.get(id) ?? 0,
cells,
}
}),
otherBytes,
}
}
async function loadPivotLabels(
rowDim: StatisticsPivotDim,
colDim: StatisticsPivotDim,
rowIds: string[],
colIds: string[],
): Promise<{ row: Map<string, string>; col: Map<string, string> }> {
const serverNames = new Map<string, string>()
const allServers = await db.select({ id: servers.id, name: servers.name, host: servers.host }).from(servers)
for (const s of allServers) serverNames.set(String(s.id), s.name || s.host)
const userNames = new Map<string, string>()
const allUsers = await db.select({ id: appUsers.id, name: appUsers.name, login: appUsers.login }).from(appUsers)
for (const u of allUsers) userNames.set(u.id, u.name || u.login)
const asnHolders = new Map<string, string>()
const asnMeta = await db.select({ asn: flowAsnMeta.asn, holder: flowAsnMeta.holder }).from(flowAsnMeta)
for (const a of asnMeta) asnHolders.set(String(a.asn), a.holder)
function label(dim: StatisticsPivotDim, id: string): string {
if (id === PIVOT_OTHER_ID) return "Прочие"
if (dim === "country") return id === "XX" ? "Неизвестно" : id
if (dim === "server") return serverNames.get(id) || id
if (dim === "user") return userNames.get(id) || id
if (dim === "asn") {
if (id === "0") return "other"
const holder = asnHolders.get(id)
return holder ? `AS${id} · ${holder}` : `AS${id}`
}
if (dim === "iface") {
const colon = id.indexOf(":")
if (colon < 0) return id
const sid = id.slice(0, colon)
const iface = id.slice(colon + 1)
const sidNum = Number(sid)
const name = Number.isFinite(sidNum) ? displayFactIface(sidNum, iface) : iface
return `${serverNames.get(sid) || sid} · ${name}`
}
return id
}
const row = new Map<string, string>()
const col = new Map<string, string>()
for (const id of rowIds) row.set(id, label(rowDim, id))
for (const id of colIds) col.set(id, label(colDim, id))
return { row, col }
}
@@ -1,5 +1,10 @@
import assert from "node:assert/strict"
import { dedupFlowRowsMaxBytes, flowTupleKey } from "./traffic-flow-dedup.js"
import {
dedupFlowRowsAcrossExporters,
dedupFlowRowsMaxBytes,
flowConversationKey,
flowTupleKey,
} from "./traffic-flow-dedup.js"
const a = {
serverId: 7,
@@ -22,4 +27,13 @@ assert.equal(flowTupleKey(a), flowTupleKey(b))
const sameIface = dedupFlowRowsMaxBytes([a, { ...a, bytes: 3_000, packets: 2 }])
assert.equal(sameIface[0]?.bytes, 15_000)
const jh = { ...a, serverId: 7, bytes: 9_000 }
const en = { ...a, serverId: 9, bytes: 11_000, inIface: "1" }
assert.equal(flowConversationKey(jh), flowConversationKey(en))
assert.notEqual(flowTupleKey(jh), flowTupleKey(en))
const across = dedupFlowRowsAcrossExporters([en, jh], (x, y) => (x.serverId === 7 ? x : y))
assert.equal(across.length, 1)
assert.equal(across[0]?.serverId, 7)
assert.equal(across[0]?.bytes, 9_000)
console.log("traffic-flow-dedup.test.ts: ok")
@@ -14,6 +14,32 @@ export function flowTupleKey(r: Pick<FlowTupleRow, "serverId" | "src" | "dst" |
return `${r.serverId}|${r.src}|${r.dst}|${r.proto}|${r.srcPort}|${r.dstPort}`
}
/** Один разговор на всех экспортёрах (JH+EN), без serverId. */
export function flowConversationKey(r: Pick<FlowTupleRow, "src" | "dst" | "proto" | "srcPort" | "dstPort">): string {
return `${r.src}|${r.dst}|${r.proto}|${r.srcPort}|${r.dstPort}`
}
/**
* Схлопнуть копии одного 5-tuple с разных серверов.
* `prefer` выбирает ряд (клиент на JH важнее голого EN).
*/
export function dedupFlowRowsAcrossExporters<T extends FlowTupleRow>(
rows: T[],
prefer: (a: T, b: T) => T,
): T[] {
const byConv = new Map<string, T>()
for (const row of rows) {
const key = flowConversationKey(row)
const prev = byConv.get(key)
if (!prev) {
byConv.set(key, row)
continue
}
byConv.set(key, prefer(prev, row))
}
return [...byConv.values()]
}
function ifaceKey(r: FlowTupleRow): string {
return `${flowTupleKey(r)}|${r.inIface}`
}
+2 -1
View File
@@ -10,6 +10,7 @@ import { resolveFlowIp } from "./traffic-flow-geoip.js"
import { invalidateTrafficFlowSettingsCache } from "./traffic-flow-settings.js"
import { isIsoCountry } from "./traffic-flow-brands.js"
import { maybeRefreshIfaces } from "./traffic-flow-ifaces.js"
import { canonicalFactIface } from "./traffic-flow-ifindex.js"
import { pickInternetPeer } from "./traffic-flow-ip.js"
import {
bumpFlowFact,
@@ -360,7 +361,7 @@ export function queueParsedFlows(serverId: number, flows: ParsedFlowInput[]): vo
bumpFlowFact({
serverId,
bucketAt: hourAt,
iface: flow.inIface,
iface: canonicalFactIface(serverId, flow.inIface),
country: country || "XX",
service: classified.service,
asn: ripe?.ok && ripe.asn ? ripe.asn : 0,
@@ -22,8 +22,9 @@ rememberServerIfaces(7, [
{ ".id": "*A", name: "wg-flow" },
{ ".id": "*D", name: "bridge" },
])
assert.equal(resolveIfaceName(7, "2").name, "ether1")
assert.equal(resolveIfaceName(7, "10").name, "wg-flow")
assert.equal(resolveIfaceName(7, "2").name, "ether1")
assert.equal(resolveIfaceName(7, "#2").name, "ether1")
assert.equal(resolveIfaceName(7, "10").name, "wg-flow")
assert.equal(resolveIfaceName(7, "13").name, "bridge")
assert.equal(resolveIfaceName(7, "0").name, "—")
assert.equal(resolveIfaceName(7, "ether1").name, "ether1")
@@ -0,0 +1,58 @@
import assert from "node:assert/strict"
import {
bindingIfaceAliases,
bindingIfaceAliasesAllServers,
canonicalFactIface,
collapseServerIfaceRows,
displayFactIface,
expandBindingIfaces,
factIfaceAliases,
rememberServerIfaces,
resetIfaceCacheForTests,
resolveIfaceName,
} from "./traffic-flow-ifindex.js"
resetIfaceCacheForTests()
assert.equal(canonicalFactIface(1, "2"), "2")
assert.deepEqual(bindingIfaceAliases(1, "gre-client"), ["gre-client"])
rememberServerIfaces(1, [{ name: "gre-client", ifindex: "2" }])
assert.equal(canonicalFactIface(1, "2"), "gre-client")
assert.equal(canonicalFactIface(1, "gre-client"), "gre-client")
assert.equal(canonicalFactIface(1, "9"), "9")
assert.equal(resolveIfaceName(1, "9").name, "#9")
assert.equal(resolveIfaceName(1, "2").name, "gre-client")
assert.equal(resolveIfaceName(1, "#2").name, "gre-client")
assert.equal(displayFactIface(1, "2"), "gre-client")
const aliases = bindingIfaceAliases(1, "gre-client")
assert.ok(aliases.includes("gre-client"))
assert.ok(aliases.includes("2"))
assert.ok(aliases.includes("#2"))
const fromIndex = factIfaceAliases("2", 1)
assert.ok(fromIndex.includes("gre-client"))
assert.ok(fromIndex.includes("2"))
assert.ok(fromIndex.includes("#2"))
const all = bindingIfaceAliasesAllServers("gre-client")
assert.ok(all.includes("2"))
const expanded = expandBindingIfaces([{ serverId: 1, iface: "gre-client" }])
assert.ok(expanded.some((x) => x.iface === "2"))
assert.ok(expanded.some((x) => x.iface === "gre-client"))
const collapsed = collapseServerIfaceRows([
{ serverId: 1, iface: "2", bytes: 10, packets: 1 },
{ serverId: 1, iface: "gre-client", bytes: 5, packets: 2 },
{ serverId: 1, iface: "wan1", bytes: 3, packets: 1 },
])
assert.equal(collapsed.length, 2)
const gre = collapsed.find((r) => r.iface === "gre-client")
assert.ok(gre)
assert.equal(gre.bytes, 15)
assert.equal(gre.packets, 3)
assert.ok(collapsed.some((r) => r.iface === "wan1"))
resetIfaceCacheForTests()
console.log("traffic-flow-ifindex.test.ts: ok")
+123 -6
View File
@@ -36,12 +36,129 @@ export function rememberServerIfaces(serverId: number, rows: RosIfaceIndexRow[])
export function resolveIfaceName(serverId: number, indexOrName: string): { name: string; index: string } {
const trimmed = String(indexOrName ?? "").trim()
if (!trimmed || trimmed === "0") return { name: "—", index: trimmed }
if (!/^\d+$/.test(trimmed)) return { name: trimmed, index: "" }
const idx = Number(trimmed)
const name = cache.get(serverId)?.get(idx)
if (name) return { name, index: trimmed }
return { name: `#${trimmed}`, index: trimmed }
const asIndex = trimmed.startsWith("#") && /^\d+$/.test(trimmed.slice(1)) ? trimmed.slice(1) : trimmed
if (!asIndex || asIndex === "0") return { name: "—", index: asIndex }
if (!/^\d+$/.test(asIndex)) return { name: trimmed, index: "" }
const name = cache.get(serverId)?.get(Number(asIndex))
if (name) return { name, index: asIndex }
return { name: `#${asIndex}`, index: asIndex }
}
/** Имя iface для факта куба: ifIndex→имя, без `#13` при пустом кэше. */
export function canonicalFactIface(serverId: number, inIface: string): string {
const trimmed = String(inIface ?? "").trim()
if (!trimmed) return trimmed
if (!/^\d+$/.test(trimmed)) return trimmed
const name = cache.get(serverId)?.get(Number(trimmed))
return name || trimmed
}
function numericIfaceIndex(iface: string): string | null {
const raw = String(iface ?? "").trim()
if (/^\d+$/.test(raw)) return raw
if (raw.startsWith("#") && /^\d+$/.test(raw.slice(1))) return raw.slice(1)
return null
}
/** Имя для UI: ifIndex → RouterOS name; `0` → «—»; miss → `#n`. */
export function displayFactIface(serverId: number, iface: string): string {
return resolveIfaceName(serverId, iface).name
}
/** Склеить факты `2` + `ether1` в одну строку после резолва ifIndex. */
export function collapseServerIfaceRows(
rows: Array<{ serverId: number; iface: string; bytes: number; packets: number }>,
): Array<{ serverId: number; iface: string; bytes: number; packets: number }> {
const acc = new Map<string, { serverId: number; iface: string; bytes: number; packets: number }>()
for (const r of rows) {
const name = displayFactIface(r.serverId, r.iface)
const k = `${r.serverId}\0${name}`
const prev = acc.get(k)
const bytes = Number(r.bytes) || 0
const packets = Number(r.packets) || 0
if (prev) {
prev.bytes += bytes
prev.packets += packets
} else {
acc.set(k, { serverId: r.serverId, iface: name, bytes, packets })
}
}
return [...acc.values()]
}
/** Ключи факта для фильтра: имя, ifIndex и `#n`. */
export function factIfaceAliases(iface: string, serverId?: number): string[] {
const raw = String(iface ?? "").trim()
if (!raw) return []
const out = new Set<string>([raw])
const idx = numericIfaceIndex(raw)
if (idx) {
out.add(idx)
out.add(`#${idx}`)
const n = Number(idx)
if (serverId != null) {
const name = cache.get(serverId)?.get(n)
if (name) out.add(name)
} else {
for (const map of cache.values()) {
const name = map.get(n)
if (name) out.add(name)
}
}
}
if (serverId != null) {
for (const a of bindingIfaceAliases(serverId, raw)) out.add(a)
} else {
for (const a of bindingIfaceAliasesAllServers(raw)) out.add(a)
}
return [...out]
}
/** Имя + ifIndex + `#n` — тот же матч, что карта `/traffic`. */
export function bindingIfaceAliases(serverId: number, interfaceName: string): string[] {
const name = String(interfaceName ?? "").trim()
if (!name) return []
const out = new Set<string>([name])
const map = cache.get(serverId)
const idx = numericIfaceIndex(name)
const canonical = (idx && map?.get(Number(idx))) || name
out.add(canonical)
if (idx) {
out.add(idx)
out.add(`#${idx}`)
}
if (!map) return [...out]
for (const [i, n] of map) {
if (n !== canonical && n !== name) continue
out.add(String(i))
out.add(`#${i}`)
}
return [...out]
}
export function bindingIfaceAliasesAllServers(interfaceName: string): string[] {
const name = String(interfaceName ?? "").trim()
const out = new Set<string>(name ? [name] : [])
for (const serverId of cache.keys()) {
for (const alias of bindingIfaceAliases(serverId, name)) out.add(alias)
}
return [...out]
}
export function expandBindingIfaces(
binds: Array<{ serverId: number; iface: string }>,
): Array<{ serverId: number; iface: string }> {
const seen = new Set<string>()
const out: Array<{ serverId: number; iface: string }> = []
for (const b of binds) {
for (const iface of bindingIfaceAliases(b.serverId, b.iface)) {
const k = `${b.serverId}\0${iface}`
if (seen.has(k)) continue
seen.add(k)
out.push({ serverId: b.serverId, iface })
}
}
return out
}
export function ifaceCacheHas(serverId: number): boolean {
@@ -584,4 +584,98 @@ try {
resetFlowCatalogForTests()
}
resetFlowRingsForTests()
resetIfaceCacheForTests()
resetRipeCacheForTests()
disableRipeEnqueueForTests()
seedFlowTopologyForTests(topo)
rememberServerIfaces(7, [
{ ".id": "*2", name: "gre-client" },
{ ".id": "*3", name: "gre-jh-en" },
])
rememberServerIfaces(9, [
{ ".id": "*1", name: "ether1" },
])
googleRipe()
ingestParsedFlowsForServerForTests(7, [payloadFlow("8.8.8.8", 12_000)])
ingestParsedFlowsForServerForTests(9, [{
src: "10.100.1.17",
dst: "8.8.8.8",
proto: 6,
srcPort: 51234,
dstPort: 443,
bytes: 12_000,
packets: 10,
inIface: "1",
outIface: "1",
nextHop: "",
}])
try {
resetFlowMapHopsCacheForTests()
const dual = await buildFlowMapHops({ minutes: 5, minSharePct: 0 })
const gre = dual.hops.find((h) => h.kind === "gre" && h.fromId === "7" && h.toId === "9")
assert.ok(gre, "GRE JH→EN сохранён")
assert.equal(gre.bytes, 12_000)
const googlePaths = (dual.servicePaths ?? []).filter((p) => p.serviceId === "svc:google")
assert.equal(googlePaths.length, 1, "один путь без копии EN")
assert.equal(googlePaths[0]?.clientId, "u1")
assert.equal(googlePaths[0]?.viaId, "7")
const googleEdge = dual.serviceEdges?.find((e) => e.toId === "svc:google" && e.fromId === "9")
assert.ok(googleEdge)
assert.equal(googleEdge.bytes, 12_000)
assert.equal(googleEdge.bps, googlePaths[0]?.bps)
} finally {
seedFlowTopologyForTests(null)
resetFlowRingsForTests()
resetIfaceCacheForTests()
resetRipeCacheForTests()
resetFlowCatalogForTests()
}
resetFlowRingsForTests()
resetIfaceCacheForTests()
resetRipeCacheForTests()
disableRipeEnqueueForTests()
seedFlowTopologyForTests(topo)
rememberServerIfaces(7, [
{ ".id": "*2", name: "gre-client" },
{ ".id": "*3", name: "gre-jh-en" },
])
googleRipe()
ingestParsedFlowsForServerForTests(7, [
payloadFlow("8.8.8.8", 8_000),
{
src: "8.8.8.8",
dst: "10.100.1.17",
proto: 6,
srcPort: 443,
dstPort: 51234,
bytes: 4_000,
packets: 8,
inIface: "3",
outIface: "2",
nextHop: "",
},
])
try {
resetFlowMapHopsCacheForTests()
const bothDir = await buildFlowMapHops({ minutes: 5, minSharePct: 0 })
const gre = bothDir.hops.find((h) => h.kind === "gre" && h.fromId === "7" && h.toId === "9")
assert.ok(gre, "GRE-hop при fwd/rev")
const googlePaths = (bothDir.servicePaths ?? []).filter((p) => p.serviceId === "svc:google")
assert.equal(googlePaths.length, 1, "fwd+rev — один клиент")
assert.equal(googlePaths[0]?.clientId, "u1")
assert.ok(!(bothDir.servicePaths ?? []).some((p) => p.serviceId === "svc:google" && p.clientId === "—"))
const googleEdge = bothDir.serviceEdges?.find((e) => e.toId === "svc:google" && e.fromId === "9")
assert.ok(googleEdge)
assert.equal(googleEdge.bytes, 12_000)
assert.equal(googleEdge.bps, googlePaths[0]?.bps)
} finally {
seedFlowTopologyForTests(null)
resetFlowRingsForTests()
resetIfaceCacheForTests()
resetRipeCacheForTests()
resetFlowCatalogForTests()
}
console.log("traffic-flow-map-hops.test.ts: ok")
+38 -4
View File
@@ -8,7 +8,7 @@ import {
mapServiceNodeId,
resolveFlowBrand,
} from "./traffic-flow-brands.js"
import { dedupFlowRowsMaxBytes } from "./traffic-flow-dedup.js"
import { dedupFlowRowsAcrossExporters, dedupFlowRowsMaxBytes } from "./traffic-flow-dedup.js"
import { getFlowListenerState, listFlowRowsForWindow } from "./traffic-flow-ingest.js"
import { resolveIfaceName } from "./traffic-flow-ifaces.js"
import { classifyFlowPlane, shouldKeepPlane } from "./traffic-flow-planes.js"
@@ -16,7 +16,7 @@ import { pickInternetPeer } from "./traffic-flow-ip.js"
import { type FlowIpMeta } from "./traffic-flow-ripe.js"
import { resolveFlowIp } from "./traffic-flow-geoip.js"
import { getTrafficFlowSettingsRow } from "./traffic-flow-settings.js"
import { loadFlowTopology, resolveClient, resolveEn, getServerCatalog } from "./traffic-flow-topology.js"
import { loadFlowTopology, resolveClient, resolveEn, getServerCatalog, type FlowTopology } from "./traffic-flow-topology.js"
import { flowDataEpoch } from "./traffic-flow-engine.js"
export const DEFAULT_MAP_SERVICE_MIN_SHARE_PCT = 5
@@ -141,6 +141,16 @@ function ifaceUsable(name: string): boolean {
return Boolean(name) && name !== "—"
}
function resolveMapClient(
topo: FlowTopology,
serverId: number,
inName: string,
outName: string,
) {
return resolveClient(topo, serverId, inName)
?? (ifaceUsable(outName) ? resolveClient(topo, serverId, outName) : null)
}
function bump(acc: Map<string, HopAcc>, key: string, seed: Omit<HopAcc, "bytes" | "bytesFwd" | "bytesRev">, bytes: number, dir: "fwd" | "rev" | "both"): void {
const prev = acc.get(key)
const addFwd = dir === "fwd" || dir === "both" ? bytes : 0
@@ -238,6 +248,21 @@ async function buildFlowMapHopsUncached(q: FlowMapHopsQuery, minSharePct: number
const enIds = new Set(topo.enNodes.map((n) => n.id))
let totalBytes = 0
function rowClient(r: (typeof working)[number]) {
const inName = resolveIfaceName(r.serverId, r.inIface).name
const outName = resolveIfaceName(r.serverId, r.outIface).name
return resolveMapClient(topo, r.serverId, inName, outName)
}
const payloadRows = wantDedup
? dedupFlowRowsAcrossExporters(working, (a, b) => {
const aCli = Boolean(rowClient(a))
const bCli = Boolean(rowClient(b))
if (aCli !== bCli) return aCli ? a : b
return a.bytes >= b.bytes ? a : b
})
: working
for (const r of working) {
const inRes = resolveIfaceName(r.serverId, r.inIface)
const outRes = resolveIfaceName(r.serverId, r.outIface)
@@ -323,10 +348,14 @@ async function buildFlowMapHopsUncached(q: FlowMapHopsQuery, minSharePct: number
}, r.bytes, "fwd")
}
}
}
for (const r of payloadRows) {
const inName = resolveIfaceName(r.serverId, r.inIface).name
const outName = resolveIfaceName(r.serverId, r.outIface).name
totalBytes += r.bytes
const peer = pickInternetPeer(r.src, r.dst, r.srcPort, r.dstPort)
const client = resolveClient(topo, r.serverId, inName)
const client = resolveMapClient(topo, r.serverId, inName, outName)
const prevDst = dstAcc.get(peer)
if (prevDst) {
prevDst.bytes += r.bytes
@@ -428,10 +457,15 @@ async function buildFlowMapHopsUncached(q: FlowMapHopsQuery, minSharePct: number
const enName = nodeName(fromId)
const viaName = nodeName(exporterId)
for (const [clientId, c] of from.clients) {
const pathKey = `${clientId}|${exporterId}|${fromId}|${toId}`
const pathKey = `${clientId}|${fromId}|${toId}`
const prevPath = svcPaths.get(pathKey)
if (prevPath) {
prevPath.bytes += c.bytes
if (exporterId !== fromId && prevPath.viaId === fromId) {
prevPath.viaId = exporterId
prevPath.viaName = viaName
}
if (prevPath.clientName === "—" && c.name !== "—") prevPath.clientName = c.name
} else {
svcPaths.set(pathKey, {
clientId,
@@ -5,7 +5,7 @@ import { Flag } from "@/components/flag"
import { Badge } from "@/components/reui/badge"
import { fmtBps, formatBytes } from "@/lib/fmt-rate"
import { cn } from "@/lib/utils"
import type { StatisticsBreakdownRow } from "@mmapp/contracts/statistics"
import { STATISTICS_UNBOUND_USER_ID, type StatisticsBreakdownRow } from "@mmapp/contracts/statistics"
export type StatisticsSliceKind = "users" | "servers" | "interfaces" | "countries" | "services" | "asns"
@@ -15,13 +15,16 @@ export function StatisticsBreakdownDataGrid({
selectedId,
onRowClick,
isLoading,
density = "full",
}: {
rows: StatisticsBreakdownRow[]
kind: StatisticsSliceKind
selectedId?: string
onRowClick?: (row: StatisticsBreakdownRow) => void
isLoading?: boolean
density?: "full" | "mini"
}) {
const mini = density === "mini"
const columns: CompactDataGridColumn<StatisticsBreakdownRow>[] = [
{
id: "label",
@@ -29,7 +32,9 @@ export function StatisticsBreakdownDataGrid({
accessorKey: "label",
cell: (row) => (
<span className={cn("flex items-center gap-2", selectedId === row.id && "font-medium")}>
{kind === "countries" && row.id !== "XX" ? <Flag code={row.id} size={16} /> : null}
{kind === "countries" && row.id !== "XX" && row.id !== STATISTICS_UNBOUND_USER_ID ? (
<Flag code={row.id} size={16} />
) : null}
<span className="truncate">{row.label || row.id}</span>
{selectedId === row.id ? (
<Badge variant="outline" size="sm">
@@ -45,18 +50,24 @@ export function StatisticsBreakdownDataGrid({
accessorKey: "bytes",
cell: (row) => <span className="tabular-nums">{formatBytes(row.bytes)}</span>,
},
{
id: "packets",
header: "Пакеты",
accessorKey: "packets",
cell: (row) => <span className="tabular-nums">{row.packets.toLocaleString("ru-RU")}</span>,
},
{
id: "bps",
header: "Средний bitrate",
accessorKey: "bps",
cell: (row) => <span className="tabular-nums">{fmtBps(row.bps)}</span>,
},
...(!mini
? [
{
id: "packets",
header: "Пакеты",
accessorKey: "packets" as const,
cell: (row: StatisticsBreakdownRow) => (
<span className="tabular-nums">{row.packets.toLocaleString("ru-RU")}</span>
),
},
{
id: "bps",
header: "Средний bitrate",
accessorKey: "bps" as const,
cell: (row: StatisticsBreakdownRow) => <span className="tabular-nums">{fmtBps(row.bps)}</span>,
},
]
: []),
{
id: "percent",
header: "Доля",
@@ -0,0 +1,57 @@
"use client"
import { Frame, FrameHeader, FramePanel, FrameTitle } from "@/components/reui/frame"
import {
StatisticsBreakdownDataGrid,
type StatisticsSliceKind,
} from "@/components/data-grids/statistics-breakdown-data-grid"
import { STATISTICS_DIMS } from "@/lib/statistics-dims"
import type { StatisticsBreakdownRow, StatisticsDto } from "@mmapp/contracts/statistics"
const MINI_ROWS = 12
function rowsForKind(data: StatisticsDto, kind: StatisticsSliceKind): StatisticsBreakdownRow[] {
if (kind === "users") return data.users
if (kind === "servers") return data.servers
if (kind === "interfaces") return data.interfaces
if (kind === "countries") return data.countries
if (kind === "services") return data.services
return data.asns
}
export function BreakdownDashboard({
data,
hidden,
selectedIdFor,
onRowClick,
isLoading,
}: {
data: StatisticsDto
hidden: Set<StatisticsSliceKind>
selectedIdFor: (kind: StatisticsSliceKind) => string | undefined
onRowClick: (kind: StatisticsSliceKind, row: StatisticsBreakdownRow) => void
isLoading?: boolean
}) {
const dims = STATISTICS_DIMS.filter((d) => !hidden.has(d.id))
return (
<div className="grid grid-cols-1 gap-4 p-4 md:grid-cols-2">
{dims.map((d) => (
<Frame key={d.id} dense>
<FrameHeader className="border-b">
<FrameTitle>{d.label}</FrameTitle>
</FrameHeader>
<FramePanel className="max-h-80 overflow-auto p-0">
<StatisticsBreakdownDataGrid
rows={rowsForKind(data, d.id).slice(0, MINI_ROWS)}
kind={d.id}
selectedId={selectedIdFor(d.id)}
onRowClick={(row) => onRowClick(d.id, row)}
isLoading={isLoading}
density="mini"
/>
</FramePanel>
</Frame>
))}
</div>
)
}
@@ -0,0 +1,73 @@
"use client"
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select"
import { STATISTICS_DIMS } from "@/lib/statistics-dims"
import type { StatisticsPivotDim } from "@mmapp/contracts/statistics"
import type { StatisticsSliceKind } from "@/components/data-grids/statistics-breakdown-data-grid"
export function DimensionSelect({
value,
onChange,
label,
}: {
value: StatisticsSliceKind
onChange: (value: StatisticsSliceKind) => void
label?: string
}) {
const current = STATISTICS_DIMS.find((d) => d.id === value)
return (
<label className="flex items-center gap-2">
{label ? <span className="text-muted-foreground text-xs whitespace-nowrap">{label}</span> : null}
<Select value={value} onValueChange={(v) => onChange(String(v ?? value) as StatisticsSliceKind)}>
<SelectTrigger size="sm" className="min-w-40">
<SelectValue>{current?.label ?? "Измерение"}</SelectValue>
</SelectTrigger>
<SelectContent align="start">
{STATISTICS_DIMS.map((d) => (
<SelectItem key={d.id} value={d.id}>
{d.label}
</SelectItem>
))}
</SelectContent>
</Select>
</label>
)
}
export function PivotDimSelect({
value,
onChange,
exclude,
label,
}: {
value: StatisticsPivotDim
onChange: (value: StatisticsPivotDim) => void
exclude?: StatisticsPivotDim
label: string
}) {
const options = STATISTICS_DIMS.filter((d) => d.pivot !== exclude)
const current = STATISTICS_DIMS.find((d) => d.pivot === value)
return (
<label className="flex items-center gap-2">
<span className="text-muted-foreground text-xs whitespace-nowrap">{label}</span>
<Select value={value} onValueChange={(v) => onChange(String(v ?? value) as StatisticsPivotDim)}>
<SelectTrigger size="sm" className="min-w-40">
<SelectValue>{current?.label ?? label}</SelectValue>
</SelectTrigger>
<SelectContent align="start">
{options.map((d) => (
<SelectItem key={d.pivot} value={d.pivot}>
{d.label}
</SelectItem>
))}
</SelectContent>
</Select>
</label>
)
}
+38
View File
@@ -0,0 +1,38 @@
"use client"
import { XIcon } from "lucide-react"
import { Badge } from "@/components/reui/badge"
import { Button } from "@/components/ui/button"
export interface SliceChip {
key: string
label: string
}
export function SliceChips({
chips,
onRemove,
}: {
chips: SliceChip[]
onRemove: (key: string) => void
}) {
if (!chips.length) return null
return (
<div className="flex flex-wrap items-center gap-1.5 border-b px-5 py-2">
{chips.map((chip) => (
<Badge key={chip.key} variant="outline" size="sm" className="gap-1 pr-0.5">
{chip.label}
<Button
type="button"
variant="ghost"
size="icon-xs"
aria-label={`Снять ${chip.label}`}
onClick={() => onRemove(chip.key)}
>
<XIcon />
</Button>
</Badge>
))}
</div>
)
}
@@ -0,0 +1,89 @@
"use client"
import { useMemo } from "react"
import { CompactDataGrid, type CompactDataGridColumn } from "@/components/data-grids/compact-data-grid"
import { formatBytes } from "@/lib/fmt-rate"
import { cn } from "@/lib/utils"
import type { StatisticsPivotDto } from "@mmapp/contracts/statistics"
interface PivotGridRow {
id: string
label: string
total: number
[key: string]: string | number
}
export function StatisticsPivotGrid({
data,
onCellClick,
isLoading,
}: {
data: StatisticsPivotDto
onCellClick?: (rowId: string, colId: string) => void
isLoading?: boolean
}) {
const rows: PivotGridRow[] = useMemo(
() =>
data.rows.map((r) => {
const next: PivotGridRow = { id: r.id, label: r.label, total: r.total }
for (const col of data.columns) {
next[`c:${col.id}`] = r.cells[col.id] ?? 0
}
return next
}),
[data],
)
const columns: CompactDataGridColumn<PivotGridRow>[] = [
{
id: "label",
header: "Измерение",
accessorKey: "label",
cell: (row) => <span className="truncate font-medium">{row.label}</span>,
},
...data.columns.map((col) => ({
id: `c:${col.id}`,
header: col.label,
accessorKey: `c:${col.id}` as const,
cell: (row: PivotGridRow) => {
const value = Number(row[`c:${col.id}`] ?? 0)
return (
<button
type="button"
className={cn(
"tabular-nums text-left hover:underline",
col.id === "__other__" || row.id === "__other__" ? "text-muted-foreground" : "",
)}
onClick={(e) => {
e.stopPropagation()
if (row.id === "__other__" || col.id === "__other__") return
onCellClick?.(row.id, col.id)
}}
>
{data.metric === "packets" ? value.toLocaleString("ru-RU") : formatBytes(value)}
</button>
)
},
})),
{
id: "total",
header: "Итого",
accessorKey: "total",
cell: (row) => (
<span className="tabular-nums font-medium">
{data.metric === "packets" ? row.total.toLocaleString("ru-RU") : formatBytes(row.total)}
</span>
),
},
]
return (
<CompactDataGrid
data={rows}
columns={columns}
isLoading={isLoading}
emptyTitle="Нет данных сводной"
emptyDescription="Выберите разные измерения строк и колонок."
/>
)
}
+31
View File
@@ -0,0 +1,31 @@
import type { StatisticsPivotDim } from "@mmapp/contracts/statistics"
import type { StatisticsSliceKind } from "@/components/data-grids/statistics-breakdown-data-grid"
export const STATISTICS_DIMS: {
id: StatisticsSliceKind
pivot: StatisticsPivotDim
label: string
}[] = [
{ id: "users", pivot: "user", label: "Пользователи" },
{ id: "servers", pivot: "server", label: "Серверы" },
{ id: "interfaces", pivot: "iface", label: "Интерфейсы" },
{ id: "countries", pivot: "country", label: "Страны" },
{ id: "services", pivot: "service", label: "Сервисы" },
{ id: "asns", pivot: "asn", label: "ASN" },
]
export function isStatisticsSliceKind(v: string): v is StatisticsSliceKind {
return STATISTICS_DIMS.some((d) => d.id === v)
}
export function isStatisticsPivotDim(v: string): v is StatisticsPivotDim {
return STATISTICS_DIMS.some((d) => d.pivot === v)
}
export function sliceKindToPivot(kind: StatisticsSliceKind): StatisticsPivotDim {
return STATISTICS_DIMS.find((d) => d.id === kind)?.pivot ?? "user"
}
export function pivotToSliceKind(dim: StatisticsPivotDim): StatisticsSliceKind {
return STATISTICS_DIMS.find((d) => d.pivot === dim)?.id ?? "users"
}
+42
View File
@@ -1,5 +1,16 @@
import { z } from "zod"
export const STATISTICS_UNBOUND_USER_ID = "__unbound__"
export const statisticsPivotDimSchema = z.enum([
"country",
"service",
"asn",
"server",
"user",
"iface",
])
export const statisticsBreakdownRowSchema = z.object({
id: z.string(),
label: z.string(),
@@ -50,8 +61,39 @@ export const statisticsDtoSchema = z.object({
asns: z.array(statisticsBreakdownRowSchema),
})
export const statisticsPivotQuerySchema = statisticsQuerySchema.extend({
row: statisticsPivotDimSchema,
col: statisticsPivotDimSchema,
metric: z.enum(["bytes", "packets"]).default("bytes"),
})
export const statisticsPivotColumnSchema = z.object({
id: z.string(),
label: z.string(),
total: z.number().nonnegative(),
})
export const statisticsPivotRowSchema = z.object({
id: z.string(),
label: z.string(),
total: z.number().nonnegative(),
cells: z.record(z.string(), z.number().nonnegative()),
})
export const statisticsPivotDtoSchema = z.object({
rowDim: statisticsPivotDimSchema,
colDim: statisticsPivotDimSchema,
metric: z.enum(["bytes", "packets"]),
columns: z.array(statisticsPivotColumnSchema),
rows: z.array(statisticsPivotRowSchema),
otherBytes: z.number().nonnegative(),
})
export type StatisticsBreakdownRow = z.infer<typeof statisticsBreakdownRowSchema>
export type StatisticsSeriesPoint = z.infer<typeof statisticsSeriesPointSchema>
export type StatisticsKpis = z.infer<typeof statisticsKpisSchema>
export type StatisticsQuery = z.infer<typeof statisticsQuerySchema>
export type StatisticsDto = z.infer<typeof statisticsDtoSchema>
export type StatisticsPivotDim = z.infer<typeof statisticsPivotDimSchema>
export type StatisticsPivotQuery = z.infer<typeof statisticsPivotQuerySchema>
export type StatisticsPivotDto = z.infer<typeof statisticsPivotDtoSchema>
+27 -2
View File
@@ -1,7 +1,13 @@
import type { StatisticsDto, StatisticsQuery } from "@mmapp/contracts/statistics"
import type {
StatisticsDto,
StatisticsPivotDto,
StatisticsPivotQuery,
StatisticsQuery,
} from "@mmapp/contracts/statistics"
import { requestJson } from "@/shared/api/http-client"
export type { StatisticsDto, StatisticsQuery }
export type { StatisticsDto, StatisticsQuery, StatisticsPivotDto, StatisticsPivotQuery }
export { STATISTICS_UNBOUND_USER_ID } from "@mmapp/contracts/statistics"
export async function getStatistics(
baseUrl: string,
@@ -18,3 +24,22 @@ export async function getStatistics(
if (query.asn != null) params.set("asn", String(query.asn))
return requestJson<StatisticsDto>(baseUrl, `/api/statistics?${params.toString()}`)
}
export async function getStatisticsPivot(
baseUrl: string,
query: StatisticsPivotQuery,
): Promise<StatisticsPivotDto> {
const params = new URLSearchParams()
params.set("from", query.from)
params.set("to", query.to)
params.set("row", query.row)
params.set("col", query.col)
params.set("metric", query.metric)
if (query.serverId != null) params.set("serverId", String(query.serverId))
if (query.userId) params.set("userId", query.userId)
if (query.iface) params.set("iface", query.iface)
if (query.country) params.set("country", query.country)
if (query.service) params.set("service", query.service)
if (query.asn != null) params.set("asn", String(query.asn))
return requestJson<StatisticsPivotDto>(baseUrl, `/api/statistics/pivot?${params.toString()}`)
}
+1 -1
View File
File diff suppressed because one or more lines are too long