Enhance API and UI for incident management and live updates
- Added a new endpoint `/api/agg/incidents` to provide a normalized snapshot of incidents for fleet triage, including severity and recommended actions. - Implemented live event streaming via `/api/live/events` for real-time updates on fleet status and incidents, enhancing observability. - Updated the Web UI to include dedicated sections for incidents and live updates, improving user navigation and access to critical information. - Enhanced API documentation to reflect new endpoints and their functionalities, ensuring clarity for developers and users.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# telemt-api
|
||||
|
||||
HTTP‑шлюз на Go для [Telemt Control API](docs/API.md): один порт, **белый список IP (CIDR)**, маршруты вида `/api/{alias}/…` → `{base_url}/v1/…`, агрегация нескольких инстансов — [`/api/agg/…`](docs/AGGREGATE.md), метрики Prometheus на `/metrics`. **Web UI** (SvelteKit) встроен в тот же процесс/образ: статика на `/`, API на `/api/…` и `/health`.
|
||||
HTTP‑шлюз на Go для [Telemt Control API](docs/API.md): один порт, **белый список IP (CIDR)**, маршруты вида `/api/{alias}/…` → `{base_url}/v1/…`, агрегация нескольких инстансов — [`/api/agg/…`](docs/AGGREGATE.md), live SSE поток — `/api/live/events`, метрики Prometheus на `/metrics`. **Web UI** (SvelteKit) встроен в тот же процесс/образ: статика на `/`, API на `/api/…` и `/health`.
|
||||
|
||||
## Быстрый старт (Linux)
|
||||
|
||||
@@ -81,6 +81,8 @@ cors_allowed_origins:
|
||||
| **[docs/API.md](docs/API.md)** | Контракт Telemt Control API (`/v1/…`) |
|
||||
| **[docs/AGGREGATE.md](docs/AGGREGATE.md)** | Агрегирующие эндпоинты шлюза (`/api/agg/…`), CORS, кэш |
|
||||
| **[docs/AGGREGATE_OPENAPI.yaml](docs/AGGREGATE_OPENAPI.yaml)** | OpenAPI 3 черновик для `/api/agg/*` (генерация типов для UI) |
|
||||
| **[docs/OPERATIONS_BASELINE.md](docs/OPERATIONS_BASELINE.md)** | Baseline UX/SLO для панели быстрого реагирования (MTTD/MTTR и критерии успеха) |
|
||||
| **[docs/INCIDENT_ROLLOUT.md](docs/INCIDENT_ROLLOUT.md)** | Пошаговый rollout incidents/live функций и настройка alert policy |
|
||||
| **[web/README.md](web/README.md)** | Web UI (SvelteKit): разработка с Vite, `PUBLIC_TELEMT_GATEWAY_URL`, встраивание в образ шлюза |
|
||||
| **[docs/GEOIP.md](docs/GEOIP.md)** | GeoLite2 City (страна/город) и опционально ASN (номер AS, организация) для IP в `unique-ips` |
|
||||
|
||||
@@ -90,6 +92,14 @@ cors_allowed_origins:
|
||||
go mod tidy && go test ./...
|
||||
```
|
||||
|
||||
## Observability (gateway)
|
||||
|
||||
Prometheus метрики доступны на `/metrics`, включая:
|
||||
|
||||
- `telemt_gateway_http_in_flight`
|
||||
- `telemt_gateway_http_requests_total{code,method,alias,endpoint}`
|
||||
- `telemt_gateway_http_request_duration_seconds{method,alias,endpoint}`
|
||||
|
||||
## CI/CD
|
||||
|
||||
В репозитории: [.gitea/workflows/docker.yaml](.gitea/workflows/docker.yaml) — тесты Go, сборка и публикация образа в Container Registry Gitea (см. раздел «Обновление и CI/CD» в [docs/GATEWAY_RUN.md](docs/GATEWAY_RUN.md)).
|
||||
|
||||
@@ -33,6 +33,7 @@
|
||||
| GET | `/api/agg/users` | Объединённый список пользователей с `by_server`, суммарным `total_megabytes` и **смерженными лимитами** (см. ниже). |
|
||||
| GET | `/api/agg/user/{username}` | Один пользователь в том же формате, что элементы `/api/agg/users` (без списка всех). Имя в пути: `[A-Za-z0-9_.-]+`. Ответ **`404`**, если пользователь не найден ни на одном успешном upstream. |
|
||||
| GET | `/api/agg/fleet-status` | По каждому алиасу: параллельно health + system/info; в `data.servers[]` — статусы подзапросов и тела `health` / `system_info` при успехе. См. [AGGREGATE_OPENAPI.yaml](AGGREGATE_OPENAPI.yaml). |
|
||||
| GET | `/api/agg/incidents` | Нормализованный snapshot инцидентов для triage-панели: `critical/warning/info`, `affected_aliases`, рекомендуемые `actions` (runbook/deep links), счётчики по severity. |
|
||||
|
||||
Все методы — **GET**; действует тот же whitelist, что и для остального API шлюза.
|
||||
|
||||
@@ -57,6 +58,18 @@
|
||||
| `include_links` | `users`, `user/…` | `true` — добавить сгенерированные `tg://proxy` ссылки (берётся первая успешная запись по пользователю). |
|
||||
| `min_total_megabytes` | `users` | Порог суммарного трафика пользователя в MiB (строго больше 0). |
|
||||
| `min_total_octets` | `users` | Устаревший вариант порога в октетах (если задан `min_total_megabytes`, он приоритетнее). |
|
||||
| `aliases` | `incidents` | Список алиасов через запятую; позволяет строить incidents snapshot по выбранной группе нод. |
|
||||
|
||||
## Live stream (SSE)
|
||||
|
||||
Для оперативного режима NOC доступен поток событий:
|
||||
|
||||
- **`GET /api/live/events`** (`text/event-stream`)
|
||||
- query: `aliases` (опционально, как в `/api/agg/*`)
|
||||
- событие: `event: snapshot`
|
||||
- payload: JSON со статусом флота (`healthy/degraded/critical`), `partial`, и массивом `incidents`
|
||||
|
||||
Поток рассчитан на UI-клиент с авто-reconnect (на фронте используется экспоненциальный backoff).
|
||||
|
||||
## Конфигурация (опционально)
|
||||
|
||||
@@ -99,4 +112,8 @@ curl -sS "http://127.0.0.1:8080/api/agg/unique-ips"
|
||||
curl -sS "http://127.0.0.1:8080/api/agg/users?include_links=false&min_total_megabytes=1"
|
||||
curl -sS "http://127.0.0.1:8080/api/agg/fleet-status"
|
||||
curl -sS "http://127.0.0.1:8080/api/agg/user/myuser?aliases=gt1"
|
||||
curl -sS "http://127.0.0.1:8080/api/agg/incidents?aliases=gt1,gt2"
|
||||
|
||||
# SSE поток snapshot-событий (пример с curl)
|
||||
curl -N "http://127.0.0.1:8080/api/live/events?aliases=gt1,gt2"
|
||||
```
|
||||
|
||||
@@ -106,6 +106,18 @@ paths:
|
||||
application/json:
|
||||
schema: { $ref: '#/components/schemas/AggEnvelopeFleetStatus' }
|
||||
|
||||
/api/agg/incidents:
|
||||
get:
|
||||
summary: Нормализованный snapshot инцидентов по флоту
|
||||
parameters:
|
||||
- $ref: '#/components/parameters/aliases'
|
||||
responses:
|
||||
'200':
|
||||
description: OK
|
||||
content:
|
||||
application/json:
|
||||
schema: { $ref: '#/components/schemas/AggEnvelopeIncidents' }
|
||||
|
||||
components:
|
||||
parameters:
|
||||
aliases:
|
||||
@@ -173,6 +185,13 @@ components:
|
||||
properties:
|
||||
data: { $ref: '#/components/schemas/FleetStatusData' }
|
||||
|
||||
AggEnvelopeIncidents:
|
||||
allOf:
|
||||
- $ref: '#/components/schemas/AggSuccessBase'
|
||||
- type: object
|
||||
properties:
|
||||
data: { $ref: '#/components/schemas/IncidentsData' }
|
||||
|
||||
TrafficRow:
|
||||
type: object
|
||||
properties:
|
||||
@@ -297,3 +316,43 @@ components:
|
||||
servers_total: { type: integer }
|
||||
servers_all_ok: { type: integer }
|
||||
servers_failed: { type: integer }
|
||||
|
||||
IncidentAction:
|
||||
type: object
|
||||
properties:
|
||||
label: { type: string }
|
||||
href: { type: string }
|
||||
|
||||
IncidentItem:
|
||||
type: object
|
||||
properties:
|
||||
id: { type: string }
|
||||
kind: { type: string }
|
||||
severity:
|
||||
type: string
|
||||
enum: [info, warning, critical]
|
||||
status:
|
||||
type: string
|
||||
enum: [firing]
|
||||
title: { type: string }
|
||||
summary: { type: string }
|
||||
affected_aliases:
|
||||
type: array
|
||||
items: { type: string }
|
||||
metric_name: { type: string }
|
||||
metric_value: { type: number, format: float }
|
||||
metric_threshold: { type: number, format: float }
|
||||
actions:
|
||||
type: array
|
||||
items: { $ref: '#/components/schemas/IncidentAction' }
|
||||
|
||||
IncidentsData:
|
||||
type: object
|
||||
properties:
|
||||
items:
|
||||
type: array
|
||||
items: { $ref: '#/components/schemas/IncidentItem' }
|
||||
total: { type: integer }
|
||||
critical_total: { type: integer }
|
||||
warning_total: { type: integer }
|
||||
info_total: { type: integer }
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# Incident Rollout Playbook
|
||||
|
||||
## Scope
|
||||
|
||||
This playbook describes staged rollout for:
|
||||
|
||||
- `/api/agg/incidents`
|
||||
- `/api/live/events` (SSE snapshots)
|
||||
- UI pages `/incidents` and `/live`
|
||||
- Live polling controls and stale indicators on key pages
|
||||
|
||||
## Stage 0 - Baseline Capture (2-3 days)
|
||||
|
||||
- Record current MTTD and MTTR from on-call logs.
|
||||
- Record manual refresh usage on main pages.
|
||||
- Save top recurring failure patterns (degraded nodes, read-only modes, bad connections spikes).
|
||||
|
||||
Outputs:
|
||||
|
||||
- baseline MTTD / MTTR
|
||||
- top 5 incident categories by frequency
|
||||
|
||||
## Stage 1 - Shadow Mode (3-5 days)
|
||||
|
||||
- Enable incidents and live pages for operators.
|
||||
- Do not change paging/escalation yet.
|
||||
- Compare incident feed against existing monitoring and mark false positives.
|
||||
|
||||
Targets:
|
||||
|
||||
- false positive ratio < 20%
|
||||
- no increase in upstream load beyond acceptable budget
|
||||
|
||||
## Stage 2 - Assisted Triage (1 week)
|
||||
|
||||
- Use `/incidents` as primary triage board.
|
||||
- Require owner + ack for active critical incidents.
|
||||
- Use runbook links from incident items.
|
||||
|
||||
Targets:
|
||||
|
||||
- ack coverage for critical incidents >= 90%
|
||||
- owner coverage for critical incidents >= 90%
|
||||
|
||||
## Stage 3 - Policy Tuning (ongoing)
|
||||
|
||||
- Adjust thresholds:
|
||||
- `bad_connections_warn` (default 1000)
|
||||
- `bad_connections_high` (default 10000)
|
||||
- Review alert fatigue weekly.
|
||||
- Promote stable thresholds into documented policy.
|
||||
|
||||
## KPI Tracking
|
||||
|
||||
- MTTD (minutes): incident first observed -> first ack
|
||||
- MTTR (minutes): incident first observed -> resolved
|
||||
- Stale time share: percentage of time live views are stale
|
||||
- Manual refresh share: manual refresh / total data update actions
|
||||
|
||||
## Fast Rollback
|
||||
|
||||
If noise or load is excessive:
|
||||
|
||||
1. disable auto-refresh by setting `refresh=0` in shared ops links
|
||||
2. switch operators back to dashboard summary only
|
||||
3. keep `/api/agg/incidents` for diagnostics while disabling SSE consumers
|
||||
|
||||
## Weekly Review Template
|
||||
|
||||
- KPI delta (MTTD, MTTR) vs baseline
|
||||
- top noisy rules
|
||||
- incidents with missing owner/ack
|
||||
- policy changes applied this week
|
||||
@@ -0,0 +1,49 @@
|
||||
# Operations Baseline (Telemt Panel)
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the baseline operating model and target SLO metrics for the Telemt response panel.
|
||||
It is used to measure impact of live updates, incidents workflow, and observability improvements.
|
||||
|
||||
## Current Baseline (Before Rollout)
|
||||
|
||||
- Dashboard refresh mode: mostly manual (`Refresh` buttons on key pages).
|
||||
- Unified incident queue: not present.
|
||||
- Alert ownership and acknowledgement flow: not present.
|
||||
- MTTR tracking: not formalized in product UI.
|
||||
- MTTD tracking: not formalized in product UI.
|
||||
- Cross-page filtering consistency: partial, per-page controls only.
|
||||
|
||||
## Baseline Risks
|
||||
|
||||
- Slow detection when operators do not refresh at the right time.
|
||||
- High context switching cost between pages during incidents.
|
||||
- No single place to triage partial/degraded node states.
|
||||
- Weak evidence trail for post-incident analysis.
|
||||
|
||||
## SLO Targets (Phase Goals)
|
||||
|
||||
- MTTD reduction: at least 30% versus baseline.
|
||||
- MTTR reduction: at least 25% versus baseline.
|
||||
- Manual refresh share during incident windows: under 10%.
|
||||
- Incident cards with `owner + ack + runbook`: over 90%.
|
||||
|
||||
## Measurement Inputs
|
||||
|
||||
- Gateway API:
|
||||
- `/api/agg/fleet-status`
|
||||
- `/api/agg/summary`
|
||||
- `/api/agg/incidents` (planned/implemented in this rollout)
|
||||
- Gateway metrics endpoint:
|
||||
- `/metrics`
|
||||
- Frontend telemetry (local panel interaction metrics):
|
||||
- refresh actions
|
||||
- ack/resolve actions
|
||||
- stale/live state durations
|
||||
|
||||
## Rollout Validation Checklist
|
||||
|
||||
- [ ] Baseline values captured before enabling auto-refresh.
|
||||
- [ ] Incident flow tested with simulated degraded upstream.
|
||||
- [ ] Alert noise review completed after first week.
|
||||
- [ ] MTTD/MTTR comparison published for phase review.
|
||||
@@ -35,6 +35,11 @@ type Handler struct {
|
||||
cache map[string]cacheEntry
|
||||
}
|
||||
|
||||
// ResolveAliasesForLive resolves aliases for external endpoints (SSE/live).
|
||||
func (h *Handler) ResolveAliasesForLive(r *http.Request) ([]string, error) {
|
||||
return h.resolveAliases(r)
|
||||
}
|
||||
|
||||
// NewHandler builds an aggregate handler; client must use a non-nil Transport (e.g. gateway shared transport).
|
||||
// Geo may be nil (no GeoLite2 lookups). cacheTTL 0 disables response caching.
|
||||
func NewHandler(p *config.Parsed, client *http.Client, geo *geoip.Service, cacheTTL time.Duration) *Handler {
|
||||
@@ -108,6 +113,8 @@ func (h *Handler) dispatch(w http.ResponseWriter, r *http.Request, sub string) {
|
||||
h.handleUsers(w, r)
|
||||
case sub == "fleet-status":
|
||||
h.handleFleetStatus(w, r)
|
||||
case sub == "incidents":
|
||||
h.handleIncidents(w, r)
|
||||
case strings.HasPrefix(sub, "user/"):
|
||||
username := strings.TrimPrefix(sub, "user/")
|
||||
if username == "" {
|
||||
@@ -304,6 +311,18 @@ func (h *Handler) handleUserOne(w http.ResponseWriter, r *http.Request, username
|
||||
writeAggOK(w, partial, row)
|
||||
}
|
||||
|
||||
func (h *Handler) handleIncidents(w http.ResponseWriter, r *http.Request) {
|
||||
aliases, err := h.resolveAliases(r)
|
||||
if err != nil {
|
||||
writeBadRequest(w, err)
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
data, partial := BuildIncidents(ctx, h, aliases)
|
||||
writeAggOK(w, partial, data)
|
||||
}
|
||||
|
||||
func writeBadRequest(w http.ResponseWriter, err error) {
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
package aggregate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type statsSummaryLite struct {
|
||||
ConnectionsBadTotal uint64 `json:"connections_bad_total"`
|
||||
ConnectionsTotal uint64 `json:"connections_total"`
|
||||
}
|
||||
|
||||
func BuildIncidents(ctx context.Context, h *Handler, aliases []string) (IncidentsData, bool) {
|
||||
out := IncidentsData{Items: make([]IncidentItem, 0)}
|
||||
partial := false
|
||||
|
||||
fleet := FetchFleetStatus(ctx, h.Client, h.Parsed, aliases)
|
||||
if fleet.ServersFailed > 0 {
|
||||
partial = true
|
||||
}
|
||||
|
||||
for _, s := range fleet.Servers {
|
||||
if !s.OK {
|
||||
severity := SeverityWarning
|
||||
if !s.HealthOK && !s.SystemInfoOK {
|
||||
severity = SeverityCritical
|
||||
}
|
||||
out.Items = append(out.Items, IncidentItem{
|
||||
ID: fmt.Sprintf("node_degraded:%s", s.Alias),
|
||||
Kind: "node_degraded",
|
||||
Severity: severity,
|
||||
Status: IncidentStatusFiring,
|
||||
Title: fmt.Sprintf("Нода %s degraded", s.Alias),
|
||||
Summary: "health/system_info не прошли полностью",
|
||||
AffectedAliases: []string{s.Alias},
|
||||
Actions: []IncidentAction{
|
||||
{Label: "Открыть ноду", Href: "/servers/" + s.Alias},
|
||||
{Label: "Runtime", Href: "/servers/" + s.Alias + "/runtime"},
|
||||
},
|
||||
})
|
||||
}
|
||||
if s.Health != nil && s.Health.ReadOnly {
|
||||
out.Items = append(out.Items, IncidentItem{
|
||||
ID: fmt.Sprintf("node_read_only:%s", s.Alias),
|
||||
Kind: "node_read_only",
|
||||
Severity: SeverityWarning,
|
||||
Status: IncidentStatusFiring,
|
||||
Title: fmt.Sprintf("Нода %s в read_only", s.Alias),
|
||||
Summary: "API ноды не принимает mutating операции",
|
||||
AffectedAliases: []string{s.Alias},
|
||||
Actions: []IncidentAction{
|
||||
{Label: "Users", Href: "/servers/" + s.Alias + "/users"},
|
||||
{Label: "Security", Href: "/servers/" + s.Alias + "/security"},
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
for _, alias := range aliases {
|
||||
data, meta := FetchTelemtGET[statsSummaryLite](ctx, h.Client, h.Parsed, alias, "stats/summary")
|
||||
if !meta.OK {
|
||||
partial = true
|
||||
continue
|
||||
}
|
||||
if data.ConnectionsBadTotal >= 10000 {
|
||||
out.Items = append(out.Items, IncidentItem{
|
||||
ID: fmt.Sprintf("bad_connections_high:%s", alias),
|
||||
Kind: "bad_connections_high",
|
||||
Severity: SeverityCritical,
|
||||
Status: IncidentStatusFiring,
|
||||
Title: fmt.Sprintf("Высокий bad connections на %s", alias),
|
||||
Summary: "Резкий рост ошибок клиентских соединений",
|
||||
AffectedAliases: []string{alias},
|
||||
MetricName: "connections_bad_total",
|
||||
MetricValue: float64(data.ConnectionsBadTotal),
|
||||
MetricThreshold: 10000,
|
||||
Actions: []IncidentAction{
|
||||
{Label: "Node dashboard", Href: "/servers/" + alias},
|
||||
},
|
||||
})
|
||||
} else if data.ConnectionsBadTotal >= 1000 {
|
||||
out.Items = append(out.Items, IncidentItem{
|
||||
ID: fmt.Sprintf("bad_connections_warn:%s", alias),
|
||||
Kind: "bad_connections_warn",
|
||||
Severity: SeverityWarning,
|
||||
Status: IncidentStatusFiring,
|
||||
Title: fmt.Sprintf("Рост bad connections на %s", alias),
|
||||
Summary: "Наблюдается рост ошибок клиентских соединений",
|
||||
AffectedAliases: []string{alias},
|
||||
MetricName: "connections_bad_total",
|
||||
MetricValue: float64(data.ConnectionsBadTotal),
|
||||
MetricThreshold: 1000,
|
||||
Actions: []IncidentAction{
|
||||
{Label: "Node dashboard", Href: "/servers/" + alias},
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
sort.Slice(out.Items, func(i, j int) bool {
|
||||
if severityWeight(out.Items[i].Severity) != severityWeight(out.Items[j].Severity) {
|
||||
return severityWeight(out.Items[i].Severity) > severityWeight(out.Items[j].Severity)
|
||||
}
|
||||
return strings.Compare(out.Items[i].ID, out.Items[j].ID) < 0
|
||||
})
|
||||
|
||||
out.Total = len(out.Items)
|
||||
for _, it := range out.Items {
|
||||
switch it.Severity {
|
||||
case SeverityCritical:
|
||||
out.CriticalTotal++
|
||||
case SeverityWarning:
|
||||
out.WarningTotal++
|
||||
default:
|
||||
out.InfoTotal++
|
||||
}
|
||||
}
|
||||
return out, partial
|
||||
}
|
||||
|
||||
func severityWeight(s IncidentSeverity) int {
|
||||
switch s {
|
||||
case SeverityCritical:
|
||||
return 3
|
||||
case SeverityWarning:
|
||||
return 2
|
||||
case SeverityInfo:
|
||||
return 1
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
func BuildLiveEnvelope(ctx context.Context, h *Handler, aliases []string) map[string]any {
|
||||
inc, partial := BuildIncidents(ctx, h, aliases)
|
||||
status := "healthy"
|
||||
if inc.CriticalTotal > 0 {
|
||||
status = "critical"
|
||||
} else if inc.WarningTotal > 0 {
|
||||
status = "degraded"
|
||||
}
|
||||
return map[string]any{
|
||||
"type": "live_snapshot",
|
||||
"timestamp": time.Now().UTC().Format(time.RFC3339Nano),
|
||||
"status": status,
|
||||
"partial": partial,
|
||||
"incidents": inc.Items,
|
||||
"counts": map[string]int{"total": inc.Total, "critical": inc.CriticalTotal, "warning": inc.WarningTotal, "info": inc.InfoTotal},
|
||||
"aliases_used": aliases,
|
||||
}
|
||||
}
|
||||
@@ -149,6 +149,52 @@ type SummaryData struct {
|
||||
TopUsersByUniqueIPs []TopUserByUniqueIPs `json:"top_users_by_unique_ips"`
|
||||
}
|
||||
|
||||
// IncidentSeverity is normalized severity used by aggregate incidents.
|
||||
type IncidentSeverity string
|
||||
|
||||
const (
|
||||
SeverityInfo IncidentSeverity = "info"
|
||||
SeverityWarning IncidentSeverity = "warning"
|
||||
SeverityCritical IncidentSeverity = "critical"
|
||||
)
|
||||
|
||||
// IncidentStatus is current calculated incident state.
|
||||
type IncidentStatus string
|
||||
|
||||
const (
|
||||
IncidentStatusFiring IncidentStatus = "firing"
|
||||
)
|
||||
|
||||
// IncidentAction points to a UI route with details/runbook context.
|
||||
type IncidentAction struct {
|
||||
Label string `json:"label"`
|
||||
Href string `json:"href"`
|
||||
}
|
||||
|
||||
// IncidentItem is one normalized fleet incident.
|
||||
type IncidentItem struct {
|
||||
ID string `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Severity IncidentSeverity `json:"severity"`
|
||||
Status IncidentStatus `json:"status"`
|
||||
Title string `json:"title"`
|
||||
Summary string `json:"summary"`
|
||||
AffectedAliases []string `json:"affected_aliases,omitempty"`
|
||||
MetricName string `json:"metric_name,omitempty"`
|
||||
MetricValue float64 `json:"metric_value,omitempty"`
|
||||
MetricThreshold float64 `json:"metric_threshold,omitempty"`
|
||||
Actions []IncidentAction `json:"actions,omitempty"`
|
||||
}
|
||||
|
||||
// IncidentsData is aggregate response payload for /api/agg/incidents.
|
||||
type IncidentsData struct {
|
||||
Items []IncidentItem `json:"items"`
|
||||
Total int `json:"total"`
|
||||
CriticalTotal int `json:"critical_total"`
|
||||
WarningTotal int `json:"warning_total"`
|
||||
InfoTotal int `json:"info_total"`
|
||||
}
|
||||
|
||||
// TopUser by summed traffic across servers for one username (мегабайты).
|
||||
type TopUser struct {
|
||||
Username string `json:"username"`
|
||||
|
||||
@@ -3,6 +3,7 @@ package server
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
@@ -181,8 +182,9 @@ func (g *Gateway) withMetrics(next http.Handler) http.Handler {
|
||||
httpInFlight.Inc()
|
||||
start := time.Now()
|
||||
alias := routeAlias(r.URL.Path)
|
||||
endpoint := routeEndpoint(r.URL.Path)
|
||||
lw := &statusWriter{ResponseWriter: w, status: http.StatusOK}
|
||||
defer observeRequest(r.Method, alias, lw.status, start)
|
||||
defer observeRequest(r.Method, alias, endpoint, lw.status, start)
|
||||
next.ServeHTTP(lw, r)
|
||||
})
|
||||
}
|
||||
@@ -206,6 +208,33 @@ func routeAlias(path string) string {
|
||||
return rest[:i]
|
||||
}
|
||||
|
||||
func routeEndpoint(path string) string {
|
||||
if path == "" || path == "/" {
|
||||
return "ui_root"
|
||||
}
|
||||
if path == "/health" || path == "/metrics" {
|
||||
return strings.TrimPrefix(path, "/")
|
||||
}
|
||||
if strings.HasPrefix(path, "/api/agg/") {
|
||||
return strings.TrimPrefix(path, "/api/agg/")
|
||||
}
|
||||
if path == "/api/agg" {
|
||||
return "agg"
|
||||
}
|
||||
if strings.HasPrefix(path, "/api/live/events") {
|
||||
return "live_events"
|
||||
}
|
||||
if strings.HasPrefix(path, "/api/") {
|
||||
rest := strings.TrimPrefix(path, "/api/")
|
||||
i := strings.IndexByte(rest, '/')
|
||||
if i < 0 {
|
||||
return "proxy_root"
|
||||
}
|
||||
return "proxy_" + rest[i+1:]
|
||||
}
|
||||
return "ui"
|
||||
}
|
||||
|
||||
func (g *Gateway) serve(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/health":
|
||||
@@ -233,6 +262,10 @@ func (g *Gateway) serve(w http.ResponseWriter, r *http.Request) {
|
||||
g.agg.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
if r.URL.Path == "/api/live/events" {
|
||||
g.serveLiveEvents(w, r)
|
||||
return
|
||||
}
|
||||
if !strings.HasPrefix(r.URL.Path, prefix) {
|
||||
g.webUI.ServeHTTP(w, r)
|
||||
return
|
||||
@@ -265,6 +298,54 @@ func (g *Gateway) serve(w http.ResponseWriter, r *http.Request) {
|
||||
rp.ServeHTTP(w, r)
|
||||
}
|
||||
|
||||
func (g *Gateway) serveLiveEvents(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
flusher, ok := w.(http.Flusher)
|
||||
if !ok {
|
||||
http.Error(w, "stream unsupported", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
aliases, err := g.agg.ResolveAliasesForLive(r)
|
||||
if err != nil {
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"ok": false,
|
||||
"error": map[string]string{"code": "bad_request", "message": err.Error()},
|
||||
})
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
w.Header().Set("Cache-Control", "no-cache")
|
||||
w.Header().Set("Connection", "keep-alive")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
|
||||
ticker := time.NewTicker(5 * time.Second)
|
||||
defer ticker.Stop()
|
||||
|
||||
push := func() {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
|
||||
defer cancel()
|
||||
payload := aggregate.BuildLiveEnvelope(ctx, g.agg, aliases)
|
||||
b, _ := json.Marshal(payload)
|
||||
_, _ = fmt.Fprintf(w, "event: snapshot\n")
|
||||
_, _ = fmt.Fprintf(w, "data: %s\n\n", string(b))
|
||||
flusher.Flush()
|
||||
}
|
||||
push()
|
||||
for {
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
push()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type statusWriter struct {
|
||||
http.ResponseWriter
|
||||
status int
|
||||
|
||||
@@ -15,17 +15,17 @@ var (
|
||||
})
|
||||
httpRequests = promauto.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: "telemt_gateway_http_requests_total",
|
||||
Help: "HTTP requests by status, method, alias.",
|
||||
}, []string{"code", "method", "alias"})
|
||||
Help: "HTTP requests by status, method, alias, endpoint.",
|
||||
}, []string{"code", "method", "alias", "endpoint"})
|
||||
httpDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
|
||||
Name: "telemt_gateway_http_request_duration_seconds",
|
||||
Help: "Request duration in seconds.",
|
||||
Buckets: prometheus.DefBuckets,
|
||||
}, []string{"method", "alias"})
|
||||
}, []string{"method", "alias", "endpoint"})
|
||||
)
|
||||
|
||||
func observeRequest(method, alias string, status int, started time.Time) {
|
||||
func observeRequest(method, alias, endpoint string, status int, started time.Time) {
|
||||
httpInFlight.Dec()
|
||||
httpRequests.WithLabelValues(strconv.Itoa(status), method, alias).Inc()
|
||||
httpDuration.WithLabelValues(method, alias).Observe(time.Since(started).Seconds())
|
||||
httpRequests.WithLabelValues(strconv.Itoa(status), method, alias, endpoint).Inc()
|
||||
httpDuration.WithLabelValues(method, alias, endpoint).Observe(time.Since(started).Seconds())
|
||||
}
|
||||
|
||||
@@ -2,6 +2,14 @@
|
||||
|
||||
SvelteKit + shadcn-svelte. В **production** статика собирается и **встраивается в образ шлюза** ([Dockerfile](../Dockerfile) в корне репозитория): панель и API на **одном порту** (например `http://127.0.0.1:8080/` — UI, `/api/…` — шлюз).
|
||||
|
||||
## Основные разделы панели
|
||||
|
||||
- `/` — обзор флота (KPI, активные IP, сводка по нодам)
|
||||
- `/users`, `/users/[username]` — пользователи и детали
|
||||
- `/ips` — unique IP + GeoIP карта
|
||||
- `/incidents` — triage-интерфейс инцидентов (`ack/resolved/owner/note` в localStorage)
|
||||
- `/live` — live snapshot (SSE, авто-reconnect)
|
||||
|
||||
## Переменная `PUBLIC_TELEMT_GATEWAY_URL`
|
||||
|
||||
| Значение | Когда |
|
||||
@@ -42,6 +50,17 @@ npm run gen:api
|
||||
|
||||
Источник: [../docs/AGGREGATE_OPENAPI.yaml](../docs/AGGREGATE_OPENAPI.yaml).
|
||||
|
||||
## URL-driven controls (операторский режим)
|
||||
|
||||
На ключевых страницах используются query-параметры:
|
||||
|
||||
- `aliases=node-a,node-b` — фильтр по нодам
|
||||
- `refresh=0|10..300` — auto-refresh (0 выключает polling)
|
||||
- `include_links=0|1` — для `/users`
|
||||
- `geo=0|1` — для `/ips`
|
||||
|
||||
Live-страница `/live` работает через SSE endpoint шлюза: `/api/live/events`.
|
||||
|
||||
## Ограничения
|
||||
|
||||
- Секреты upstream к Telemt задаются на шлюзе (`authorization_env`), не в браузере.
|
||||
|
||||
@@ -20,6 +20,37 @@ export type AggEnvelope<T> = {
|
||||
data: T;
|
||||
};
|
||||
|
||||
export type IncidentSeverity = 'info' | 'warning' | 'critical';
|
||||
|
||||
export type IncidentStatus = 'firing';
|
||||
|
||||
export type IncidentAction = {
|
||||
label: string;
|
||||
href: string;
|
||||
};
|
||||
|
||||
export type IncidentItem = {
|
||||
id: string;
|
||||
kind: string;
|
||||
severity: IncidentSeverity;
|
||||
status: IncidentStatus;
|
||||
title: string;
|
||||
summary: string;
|
||||
affected_aliases?: string[];
|
||||
metric_name?: string;
|
||||
metric_value?: number;
|
||||
metric_threshold?: number;
|
||||
actions?: IncidentAction[];
|
||||
};
|
||||
|
||||
export type IncidentsData = {
|
||||
items: IncidentItem[];
|
||||
total: number;
|
||||
critical_total: number;
|
||||
warning_total: number;
|
||||
info_total: number;
|
||||
};
|
||||
|
||||
export class ApiError extends Error {
|
||||
constructor(
|
||||
message: string,
|
||||
@@ -120,6 +151,23 @@ export async function fetchAggUser(
|
||||
return body as AggEnvelope<components['schemas']['UsersRow']>;
|
||||
}
|
||||
|
||||
export async function fetchAggIncidents(params?: { aliases?: string }): Promise<AggEnvelope<IncidentsData>> {
|
||||
const q = new URLSearchParams();
|
||||
if (params?.aliases) q.set('aliases', params.aliases);
|
||||
const url = `${gatewayBase()}/api/agg/incidents${q.toString() ? `?${q}` : ''}`;
|
||||
const res = await fetch(url);
|
||||
const body = (await parseJson(res)) as Record<string, unknown> | null;
|
||||
if (!res.ok) throw new ApiError(`incidents HTTP ${res.status}`, res.status, body);
|
||||
if (!body || body.ok !== true) throw new ApiError('incidents: ok !== true', res.status, body);
|
||||
return body as AggEnvelope<IncidentsData>;
|
||||
}
|
||||
|
||||
export function liveEventsUrl(params?: { aliases?: string }): string {
|
||||
const q = new URLSearchParams();
|
||||
if (params?.aliases) q.set('aliases', params.aliases);
|
||||
return `${gatewayBase()}/api/live/events${q.toString() ? `?${q}` : ''}`;
|
||||
}
|
||||
|
||||
/** Путь к upstream без префикса /v1 — шлюз сам добавляет path_prefix. */
|
||||
function apiUrl(alias: string, path: string): string {
|
||||
const p = path.replace(/^\/+/, '');
|
||||
|
||||
@@ -9,6 +9,8 @@
|
||||
import NetworkIcon from '@lucide/svelte/icons/network';
|
||||
import SettingsIcon from '@lucide/svelte/icons/settings';
|
||||
import RefreshCwIcon from '@lucide/svelte/icons/refresh-cw';
|
||||
import SirenIcon from '@lucide/svelte/icons/siren';
|
||||
import RadioIcon from '@lucide/svelte/icons/radio';
|
||||
import * as Sidebar from '$lib/components/ui/sidebar/index.js';
|
||||
|
||||
let {
|
||||
@@ -76,6 +78,26 @@
|
||||
{/snippet}
|
||||
</Sidebar.MenuButton>
|
||||
</Sidebar.MenuItem>
|
||||
<Sidebar.MenuItem>
|
||||
<Sidebar.MenuButton isActive={active('/incidents')} tooltipContent="Incidents">
|
||||
{#snippet child({ props })}
|
||||
<a href="/incidents" {...props}>
|
||||
<SirenIcon />
|
||||
<span>Инциденты</span>
|
||||
</a>
|
||||
{/snippet}
|
||||
</Sidebar.MenuButton>
|
||||
</Sidebar.MenuItem>
|
||||
<Sidebar.MenuItem>
|
||||
<Sidebar.MenuButton isActive={active('/live')} tooltipContent="Live">
|
||||
{#snippet child({ props })}
|
||||
<a href="/live" {...props}>
|
||||
<RadioIcon />
|
||||
<span>Live</span>
|
||||
</a>
|
||||
{/snippet}
|
||||
</Sidebar.MenuButton>
|
||||
</Sidebar.MenuItem>
|
||||
</Sidebar.Menu>
|
||||
</Sidebar.Group>
|
||||
{#if serverAliases.length > 0}
|
||||
|
||||
+114
-10
@@ -1,5 +1,7 @@
|
||||
<script lang="ts">
|
||||
import { onMount } from 'svelte';
|
||||
import { page } from '$app/state';
|
||||
import { goto } from '$app/navigation';
|
||||
import {
|
||||
ApiError,
|
||||
fetchAggFleetStatus,
|
||||
@@ -73,6 +75,14 @@
|
||||
let uniqueIps = $state<components['schemas']['UniqueIPsRow'][] | null>(null);
|
||||
let generatedAt = $state<string | null>(null);
|
||||
let partial = $state(false);
|
||||
let aliases = $state('');
|
||||
let aliasesInput = $state('');
|
||||
let refreshSeconds = $state(30);
|
||||
let refreshInput = $state('30');
|
||||
let lastSuccessAtMs = $state<number | null>(null);
|
||||
let nowMs = $state(Date.now());
|
||||
let pollTimer: ReturnType<typeof setInterval> | null = null;
|
||||
let staleTimer: ReturnType<typeof setInterval> | null = null;
|
||||
let nodeStats = $state<
|
||||
Record<
|
||||
string,
|
||||
@@ -85,24 +95,25 @@
|
||||
err = null;
|
||||
try {
|
||||
const [s, f, u] = await Promise.all([
|
||||
fetchAggSummary({ top_n: 15 }),
|
||||
fetchAggFleetStatus(),
|
||||
fetchAggUniqueIps({ geo: true })
|
||||
fetchAggSummary({ top_n: 15, aliases: aliases || undefined }),
|
||||
fetchAggFleetStatus({ aliases: aliases || undefined }),
|
||||
fetchAggUniqueIps({ aliases: aliases || undefined, geo: true })
|
||||
]);
|
||||
generatedAt = s.generated_at;
|
||||
partial = !!(s.partial || f.partial || u.partial);
|
||||
summary = s.data;
|
||||
fleet = f.data ?? null;
|
||||
uniqueIps = u.data ?? null;
|
||||
lastSuccessAtMs = Date.now();
|
||||
|
||||
// Как бейдж «OK» в таблице: только health + system/info (без x.ok — в JSON поле ok опционально).
|
||||
const aliases = (fleet?.servers ?? [])
|
||||
const okAliases = (fleet?.servers ?? [])
|
||||
.filter((x) => x.alias && x.health_ok && x.system_info_ok)
|
||||
.map((x) => x.alias as string);
|
||||
const next: typeof nodeStats = {};
|
||||
const concurrency = 4;
|
||||
for (let i = 0; i < aliases.length; i += concurrency) {
|
||||
const batch = aliases.slice(i, i + concurrency);
|
||||
for (let i = 0; i < okAliases.length; i += concurrency) {
|
||||
const batch = okAliases.slice(i, i + concurrency);
|
||||
await Promise.all(
|
||||
batch.map(async (alias) => {
|
||||
try {
|
||||
@@ -130,7 +141,70 @@
|
||||
}
|
||||
}
|
||||
|
||||
onMount(load);
|
||||
function parseRefresh(raw: string | null): number {
|
||||
if (raw == null || raw.trim() === '') return 30;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n)) return 30;
|
||||
if (n === 0) return 0;
|
||||
return Math.max(10, Math.min(300, Math.floor(n)));
|
||||
}
|
||||
|
||||
function restartPolling() {
|
||||
if (pollTimer) {
|
||||
clearInterval(pollTimer);
|
||||
pollTimer = null;
|
||||
}
|
||||
if (refreshSeconds > 0) {
|
||||
pollTimer = setInterval(() => {
|
||||
void load();
|
||||
}, refreshSeconds * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
async function applyControls() {
|
||||
const q = new URLSearchParams(page.url.searchParams);
|
||||
const normalizedAliases = aliasesInput.trim();
|
||||
const parsedRefresh = parseRefresh(refreshInput);
|
||||
if (normalizedAliases) q.set('aliases', normalizedAliases);
|
||||
else q.delete('aliases');
|
||||
if (parsedRefresh === 30) q.delete('refresh');
|
||||
else q.set('refresh', String(parsedRefresh));
|
||||
const qs = q.toString();
|
||||
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
|
||||
replaceState: true,
|
||||
noScroll: true,
|
||||
keepFocus: true
|
||||
});
|
||||
}
|
||||
|
||||
let queryKey = $derived(page.url.searchParams.toString());
|
||||
$effect(() => {
|
||||
queryKey;
|
||||
if (typeof window === 'undefined') return;
|
||||
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
|
||||
refreshSeconds = parseRefresh(page.url.searchParams.get('refresh'));
|
||||
aliasesInput = aliases;
|
||||
refreshInput = String(refreshSeconds);
|
||||
restartPolling();
|
||||
void load();
|
||||
});
|
||||
|
||||
let staleSeconds = $derived.by(() =>
|
||||
lastSuccessAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastSuccessAtMs) / 1000))
|
||||
);
|
||||
let isStale = $derived.by(() =>
|
||||
staleSeconds == null ? true : staleSeconds > (refreshSeconds > 0 ? refreshSeconds * 2 : 60)
|
||||
);
|
||||
|
||||
onMount(() => {
|
||||
staleTimer = setInterval(() => {
|
||||
nowMs = Date.now();
|
||||
}, 1000);
|
||||
return () => {
|
||||
if (pollTimer) clearInterval(pollTimer);
|
||||
if (staleTimer) clearInterval(staleTimer);
|
||||
};
|
||||
});
|
||||
|
||||
/** Число из ответа Telemt (поле может отсутствовать или прийти строкой). */
|
||||
function numStat(v: unknown, fallback = 0): number {
|
||||
@@ -284,9 +358,39 @@
|
||||
</p>
|
||||
{/if}
|
||||
</div>
|
||||
<Button variant="outline" size="sm" onclick={load} disabled={loading}>
|
||||
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
|
||||
Обновить
|
||||
<div class="flex items-center gap-2">
|
||||
<Badge variant={isStale ? 'secondary' : 'default'}>
|
||||
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
|
||||
</Badge>
|
||||
<Button variant="outline" size="sm" onclick={() => void load()} disabled={loading}>
|
||||
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
|
||||
Обновить
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="mb-4 flex flex-wrap items-end gap-2">
|
||||
<label class="text-xs text-muted-foreground">
|
||||
Aliases
|
||||
<input
|
||||
type="text"
|
||||
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
|
||||
bind:value={aliasesInput}
|
||||
placeholder="node-a,node-b"
|
||||
/>
|
||||
</label>
|
||||
<label class="text-xs text-muted-foreground">
|
||||
Refresh (sec)
|
||||
<input
|
||||
type="number"
|
||||
min="0"
|
||||
max="300"
|
||||
class="mt-1 h-9 w-28 rounded-md border bg-background px-2 text-sm"
|
||||
bind:value={refreshInput}
|
||||
/>
|
||||
</label>
|
||||
<Button variant="outline" size="sm" onclick={() => void applyControls()} disabled={loading}>
|
||||
Apply
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -0,0 +1,351 @@
|
||||
<script lang="ts">
|
||||
import { onMount } from 'svelte';
|
||||
import { page } from '$app/state';
|
||||
import { goto } from '$app/navigation';
|
||||
import {
|
||||
ApiError,
|
||||
fetchAggIncidents,
|
||||
type IncidentItem,
|
||||
type IncidentsData
|
||||
} from '$lib/api/client.js';
|
||||
import * as Card from '$lib/components/ui/card/index.js';
|
||||
import * as Table from '$lib/components/ui/table/index.js';
|
||||
import { Alert, AlertDescription, AlertTitle } from '$lib/components/ui/alert/index.js';
|
||||
import { Badge } from '$lib/components/ui/badge/index.js';
|
||||
import { Button } from '$lib/components/ui/button/index.js';
|
||||
import RefreshCwIcon from '@lucide/svelte/icons/refresh-cw';
|
||||
|
||||
type TriageState = {
|
||||
ack?: boolean;
|
||||
resolved?: boolean;
|
||||
owner?: string;
|
||||
note?: string;
|
||||
};
|
||||
|
||||
const TRIAGE_KEY = 'telemt.incidents.triage.v1';
|
||||
|
||||
let loading = $state(true);
|
||||
let err = $state<string | null>(null);
|
||||
let partial = $state(false);
|
||||
let generatedAt = $state<string | null>(null);
|
||||
let data = $state<IncidentsData | null>(null);
|
||||
let aliases = $state('');
|
||||
let aliasesInput = $state('');
|
||||
let refreshSeconds = $state(30);
|
||||
let refreshInput = $state('30');
|
||||
let lastSuccessAtMs = $state<number | null>(null);
|
||||
let nowMs = $state(Date.now());
|
||||
let pollTimer: ReturnType<typeof setInterval> | null = null;
|
||||
let staleTimer: ReturnType<typeof setInterval> | null = null;
|
||||
let triageById = $state<Record<string, TriageState>>({});
|
||||
|
||||
function parseRefresh(raw: string | null): number {
|
||||
if (raw == null || raw.trim() === '') return 30;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n)) return 30;
|
||||
if (n === 0) return 0;
|
||||
return Math.max(10, Math.min(300, Math.floor(n)));
|
||||
}
|
||||
|
||||
function triageFor(id: string): TriageState {
|
||||
return triageById[id] ?? {};
|
||||
}
|
||||
|
||||
function saveTriage() {
|
||||
if (typeof localStorage === 'undefined') return;
|
||||
localStorage.setItem(TRIAGE_KEY, JSON.stringify(triageById));
|
||||
}
|
||||
|
||||
function loadTriage() {
|
||||
if (typeof localStorage === 'undefined') return;
|
||||
try {
|
||||
const raw = localStorage.getItem(TRIAGE_KEY);
|
||||
if (!raw) return;
|
||||
const parsed = JSON.parse(raw) as Record<string, TriageState>;
|
||||
if (parsed && typeof parsed === 'object') triageById = parsed;
|
||||
} catch {
|
||||
triageById = {};
|
||||
}
|
||||
}
|
||||
|
||||
function patchTriage(id: string, patch: Partial<TriageState>) {
|
||||
triageById = {
|
||||
...triageById,
|
||||
[id]: {
|
||||
...triageFor(id),
|
||||
...patch
|
||||
}
|
||||
};
|
||||
saveTriage();
|
||||
}
|
||||
|
||||
function toggleAck(id: string) {
|
||||
const t = triageFor(id);
|
||||
patchTriage(id, { ack: !t.ack });
|
||||
}
|
||||
|
||||
function toggleResolved(id: string) {
|
||||
const t = triageFor(id);
|
||||
patchTriage(id, { resolved: !t.resolved });
|
||||
}
|
||||
|
||||
function severityVariant(sev: IncidentItem['severity']): 'default' | 'secondary' | 'destructive' | 'outline' {
|
||||
if (sev === 'critical') return 'destructive';
|
||||
if (sev === 'warning') return 'secondary';
|
||||
return 'outline';
|
||||
}
|
||||
|
||||
function restartPolling() {
|
||||
if (pollTimer) {
|
||||
clearInterval(pollTimer);
|
||||
pollTimer = null;
|
||||
}
|
||||
if (refreshSeconds > 0) {
|
||||
pollTimer = setInterval(() => {
|
||||
void load();
|
||||
}, refreshSeconds * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
async function load() {
|
||||
loading = true;
|
||||
err = null;
|
||||
try {
|
||||
const env = await fetchAggIncidents({ aliases: aliases || undefined });
|
||||
data = env.data ?? null;
|
||||
partial = !!env.partial;
|
||||
generatedAt = env.generated_at;
|
||||
lastSuccessAtMs = Date.now();
|
||||
} catch (e) {
|
||||
err = e instanceof ApiError ? e.message : String(e);
|
||||
} finally {
|
||||
loading = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function applyControls() {
|
||||
const q = new URLSearchParams(page.url.searchParams);
|
||||
const normalizedAliases = aliasesInput.trim();
|
||||
const parsedRefresh = parseRefresh(refreshInput);
|
||||
if (normalizedAliases) q.set('aliases', normalizedAliases);
|
||||
else q.delete('aliases');
|
||||
if (parsedRefresh === 30) q.delete('refresh');
|
||||
else q.set('refresh', String(parsedRefresh));
|
||||
const qs = q.toString();
|
||||
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
|
||||
replaceState: true,
|
||||
noScroll: true,
|
||||
keepFocus: true
|
||||
});
|
||||
}
|
||||
|
||||
let queryKey = $derived(page.url.searchParams.toString());
|
||||
$effect(() => {
|
||||
queryKey;
|
||||
if (typeof window === 'undefined') return;
|
||||
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
|
||||
refreshSeconds = parseRefresh(page.url.searchParams.get('refresh'));
|
||||
aliasesInput = aliases;
|
||||
refreshInput = String(refreshSeconds);
|
||||
restartPolling();
|
||||
void load();
|
||||
});
|
||||
|
||||
let staleSeconds = $derived.by(() =>
|
||||
lastSuccessAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastSuccessAtMs) / 1000))
|
||||
);
|
||||
let isStale = $derived.by(() =>
|
||||
staleSeconds == null ? true : staleSeconds > (refreshSeconds > 0 ? refreshSeconds * 2 : 60)
|
||||
);
|
||||
|
||||
onMount(() => {
|
||||
loadTriage();
|
||||
staleTimer = setInterval(() => {
|
||||
nowMs = Date.now();
|
||||
}, 1000);
|
||||
return () => {
|
||||
if (pollTimer) clearInterval(pollTimer);
|
||||
if (staleTimer) clearInterval(staleTimer);
|
||||
};
|
||||
});
|
||||
</script>
|
||||
|
||||
<div class="mb-6 flex flex-wrap items-center justify-between gap-4">
|
||||
<div>
|
||||
<h1 class="text-2xl font-semibold tracking-tight">Инциденты</h1>
|
||||
<p class="text-sm text-muted-foreground">Сводка и triage по всему флоту.</p>
|
||||
{#if generatedAt}
|
||||
<p class="mt-1 text-xs text-muted-foreground">Снимок: {new Date(generatedAt).toLocaleString()}</p>
|
||||
{/if}
|
||||
</div>
|
||||
<div class="flex items-center gap-2">
|
||||
<Badge variant={isStale ? 'secondary' : 'default'}>
|
||||
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
|
||||
</Badge>
|
||||
<Button variant="outline" size="sm" onclick={() => void load()} disabled={loading}>
|
||||
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
|
||||
Обновить
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="mb-4 flex flex-wrap items-end gap-2">
|
||||
<label class="text-xs text-muted-foreground">
|
||||
Aliases
|
||||
<input
|
||||
type="text"
|
||||
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
|
||||
bind:value={aliasesInput}
|
||||
placeholder="node-a,node-b"
|
||||
/>
|
||||
</label>
|
||||
<label class="text-xs text-muted-foreground">
|
||||
Refresh (sec)
|
||||
<input
|
||||
type="number"
|
||||
min="0"
|
||||
max="300"
|
||||
class="mt-1 h-9 w-28 rounded-md border bg-background px-2 text-sm"
|
||||
bind:value={refreshInput}
|
||||
/>
|
||||
</label>
|
||||
<Button variant="outline" size="sm" onclick={() => void applyControls()} disabled={loading}>
|
||||
Apply
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{#if partial}
|
||||
<Alert class="mb-4">
|
||||
<AlertTitle>Частичные данные</AlertTitle>
|
||||
<AlertDescription>Не все upstream ответили успешно.</AlertDescription>
|
||||
</Alert>
|
||||
{/if}
|
||||
|
||||
{#if err}
|
||||
<Alert variant="destructive">
|
||||
<AlertTitle>Ошибка</AlertTitle>
|
||||
<AlertDescription>{err}</AlertDescription>
|
||||
</Alert>
|
||||
{:else}
|
||||
<div class="mb-4 grid gap-4 sm:grid-cols-3">
|
||||
<Card.Root>
|
||||
<Card.Header class="pb-2">
|
||||
<Card.Description>Critical</Card.Description>
|
||||
<Card.Title class="text-2xl text-red-500 tabular-nums">{data?.critical_total ?? 0}</Card.Title>
|
||||
</Card.Header>
|
||||
</Card.Root>
|
||||
<Card.Root>
|
||||
<Card.Header class="pb-2">
|
||||
<Card.Description>Warning</Card.Description>
|
||||
<Card.Title class="text-2xl text-amber-500 tabular-nums">{data?.warning_total ?? 0}</Card.Title>
|
||||
</Card.Header>
|
||||
</Card.Root>
|
||||
<Card.Root>
|
||||
<Card.Header class="pb-2">
|
||||
<Card.Description>Info</Card.Description>
|
||||
<Card.Title class="text-2xl tabular-nums">{data?.info_total ?? 0}</Card.Title>
|
||||
</Card.Header>
|
||||
</Card.Root>
|
||||
</div>
|
||||
|
||||
<Card.Root>
|
||||
<Card.Content class="p-0">
|
||||
<Table.Root>
|
||||
<Table.Header>
|
||||
<Table.Row>
|
||||
<Table.Head>Severity</Table.Head>
|
||||
<Table.Head>Инцидент</Table.Head>
|
||||
<Table.Head>Серверы</Table.Head>
|
||||
<Table.Head>Runbook</Table.Head>
|
||||
<Table.Head>Triage</Table.Head>
|
||||
</Table.Row>
|
||||
</Table.Header>
|
||||
<Table.Body>
|
||||
{#if (data?.items.length ?? 0) === 0}
|
||||
<Table.Row>
|
||||
<Table.Cell colspan={5} class="p-4 text-center text-muted-foreground">
|
||||
Инцидентов нет
|
||||
</Table.Cell>
|
||||
</Table.Row>
|
||||
{:else}
|
||||
{#each data?.items ?? [] as item (item.id)}
|
||||
<Table.Row>
|
||||
<Table.Cell>
|
||||
<Badge variant={severityVariant(item.severity)}>{item.severity}</Badge>
|
||||
</Table.Cell>
|
||||
<Table.Cell class="max-w-[360px]">
|
||||
<div class="font-medium">{item.title}</div>
|
||||
<div class="text-xs text-muted-foreground">{item.summary}</div>
|
||||
{#if item.metric_name}
|
||||
<div class="mt-1 text-xs text-muted-foreground">
|
||||
{item.metric_name}: {item.metric_value} / {item.metric_threshold}
|
||||
</div>
|
||||
{/if}
|
||||
</Table.Cell>
|
||||
<Table.Cell>
|
||||
<div class="flex flex-wrap gap-1">
|
||||
{#each item.affected_aliases ?? [] as alias (alias)}
|
||||
<Badge variant="outline">{alias}</Badge>
|
||||
{/each}
|
||||
</div>
|
||||
</Table.Cell>
|
||||
<Table.Cell class="max-w-[240px]">
|
||||
<div class="flex flex-wrap gap-1">
|
||||
{#each item.actions ?? [] as action (action.label + action.href)}
|
||||
<a href={action.href} class="text-primary hover:underline text-xs">
|
||||
{action.label}
|
||||
</a>
|
||||
{/each}
|
||||
</div>
|
||||
</Table.Cell>
|
||||
<Table.Cell class="min-w-[280px]">
|
||||
{@const triage = triageFor(item.id)}
|
||||
<div class="mb-2 flex flex-wrap gap-1">
|
||||
<Button
|
||||
size="xs"
|
||||
variant={triage.ack ? 'default' : 'outline'}
|
||||
onclick={() => toggleAck(item.id)}
|
||||
>
|
||||
{triage.ack ? 'unack' : 'ack'}
|
||||
</Button>
|
||||
<Button
|
||||
size="xs"
|
||||
variant={triage.resolved ? 'default' : 'outline'}
|
||||
onclick={() => toggleResolved(item.id)}
|
||||
>
|
||||
{triage.resolved ? 'unresolve' : 'resolve'}
|
||||
</Button>
|
||||
</div>
|
||||
<div class="mb-1">
|
||||
<input
|
||||
type="text"
|
||||
class="h-8 w-full rounded-md border bg-background px-2 text-xs"
|
||||
placeholder="owner"
|
||||
value={triage.owner ?? ''}
|
||||
oninput={(e) =>
|
||||
patchTriage(item.id, {
|
||||
owner: (e.currentTarget as HTMLInputElement).value
|
||||
})}
|
||||
/>
|
||||
</div>
|
||||
<div>
|
||||
<input
|
||||
type="text"
|
||||
class="h-8 w-full rounded-md border bg-background px-2 text-xs"
|
||||
placeholder="note"
|
||||
value={triage.note ?? ''}
|
||||
oninput={(e) =>
|
||||
patchTriage(item.id, {
|
||||
note: (e.currentTarget as HTMLInputElement).value
|
||||
})}
|
||||
/>
|
||||
</div>
|
||||
</Table.Cell>
|
||||
</Table.Row>
|
||||
{/each}
|
||||
{/if}
|
||||
</Table.Body>
|
||||
</Table.Root>
|
||||
</Card.Content>
|
||||
</Card.Root>
|
||||
{/if}
|
||||
@@ -1,5 +1,7 @@
|
||||
<script lang="ts">
|
||||
import { onMount, tick } from 'svelte';
|
||||
import { page } from '$app/state';
|
||||
import { goto } from '$app/navigation';
|
||||
import { fetchAggUniqueIps, ApiError } from '$lib/api/client.js';
|
||||
import type { components } from '$lib/api/aggregate.gen.js';
|
||||
import type * as Leaflet from 'leaflet';
|
||||
@@ -17,6 +19,14 @@
|
||||
let rows = $state<components['schemas']['UniqueIPsRow'][]>([]);
|
||||
let partial = $state(false);
|
||||
let geo = $state(true);
|
||||
let aliases = $state('');
|
||||
let aliasesInput = $state('');
|
||||
let refreshSeconds = $state(30);
|
||||
let refreshInput = $state('30');
|
||||
let lastSuccessAtMs = $state<number | null>(null);
|
||||
let nowMs = $state(Date.now());
|
||||
let pollTimer: ReturnType<typeof setInterval> | null = null;
|
||||
let staleTimer: ReturnType<typeof setInterval> | null = null;
|
||||
|
||||
let mapEl = $state<HTMLDivElement | null>(null);
|
||||
let leaflet: typeof import('leaflet') | null = null;
|
||||
@@ -273,9 +283,10 @@
|
||||
// Ждём, пока bind:this завершится и контейнеру карты будет доступен размер.
|
||||
await tick();
|
||||
initMap();
|
||||
const env = await fetchAggUniqueIps({ geo });
|
||||
const env = await fetchAggUniqueIps({ aliases: aliases || undefined, geo });
|
||||
partial = !!env.partial;
|
||||
rows = env.data ?? [];
|
||||
lastSuccessAtMs = Date.now();
|
||||
} catch (e) {
|
||||
err = e instanceof ApiError ? e.message : String(e);
|
||||
} finally {
|
||||
@@ -288,11 +299,77 @@
|
||||
}
|
||||
}
|
||||
|
||||
onMount(() => {
|
||||
// Инициализация Leaflet и загрузка данных происходит в `load()`.
|
||||
// Это избегает проблем с SSR (если он включён).
|
||||
function parseRefresh(raw: string | null): number {
|
||||
if (raw == null || raw.trim() === '') return 30;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n)) return 30;
|
||||
if (n === 0) return 0;
|
||||
return Math.max(10, Math.min(300, Math.floor(n)));
|
||||
}
|
||||
|
||||
function parse01(raw: string | null, fallback: boolean): boolean {
|
||||
if (raw == null || raw === '') return fallback;
|
||||
return raw === '1';
|
||||
}
|
||||
|
||||
function restartPolling() {
|
||||
if (pollTimer) {
|
||||
clearInterval(pollTimer);
|
||||
pollTimer = null;
|
||||
}
|
||||
if (refreshSeconds > 0) {
|
||||
pollTimer = setInterval(() => {
|
||||
void load();
|
||||
}, refreshSeconds * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
async function applyControls() {
|
||||
const q = new URLSearchParams(page.url.searchParams);
|
||||
const normalizedAliases = aliasesInput.trim();
|
||||
const parsedRefresh = parseRefresh(refreshInput);
|
||||
if (normalizedAliases) q.set('aliases', normalizedAliases);
|
||||
else q.delete('aliases');
|
||||
if (parsedRefresh === 30) q.delete('refresh');
|
||||
else q.set('refresh', String(parsedRefresh));
|
||||
q.set('geo', geo ? '1' : '0');
|
||||
const qs = q.toString();
|
||||
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
|
||||
replaceState: true,
|
||||
noScroll: true,
|
||||
keepFocus: true
|
||||
});
|
||||
}
|
||||
|
||||
let queryKey = $derived(page.url.searchParams.toString());
|
||||
$effect(() => {
|
||||
queryKey;
|
||||
if (typeof window === 'undefined') return;
|
||||
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
|
||||
refreshSeconds = parseRefresh(page.url.searchParams.get('refresh'));
|
||||
geo = parse01(page.url.searchParams.get('geo'), true);
|
||||
aliasesInput = aliases;
|
||||
refreshInput = String(refreshSeconds);
|
||||
restartPolling();
|
||||
void load();
|
||||
});
|
||||
|
||||
let staleSeconds = $derived.by(() =>
|
||||
lastSuccessAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastSuccessAtMs) / 1000))
|
||||
);
|
||||
let isStale = $derived.by(() =>
|
||||
staleSeconds == null ? true : staleSeconds > (refreshSeconds > 0 ? refreshSeconds * 2 : 60)
|
||||
);
|
||||
|
||||
onMount(() => {
|
||||
staleTimer = setInterval(() => {
|
||||
nowMs = Date.now();
|
||||
}, 1000);
|
||||
return () => {
|
||||
if (pollTimer) clearInterval(pollTimer);
|
||||
if (staleTimer) clearInterval(staleTimer);
|
||||
};
|
||||
});
|
||||
</script>
|
||||
|
||||
<div class="mb-6 flex flex-wrap items-end justify-between gap-4">
|
||||
@@ -301,17 +378,50 @@
|
||||
<p class="text-sm text-muted-foreground">Снимок active/recent с шлюза; GeoIP из конфига шлюза.</p>
|
||||
</div>
|
||||
<div class="flex items-center gap-2">
|
||||
<Badge variant={isStale ? 'secondary' : 'default'}>
|
||||
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
|
||||
</Badge>
|
||||
<label class="flex cursor-pointer items-center gap-2 text-sm">
|
||||
<input type="checkbox" bind:checked={geo} onchange={load} class="rounded border" />
|
||||
<input
|
||||
type="checkbox"
|
||||
bind:checked={geo}
|
||||
onchange={() => void applyControls()}
|
||||
class="rounded border"
|
||||
/>
|
||||
GeoIP
|
||||
</label>
|
||||
<Button variant="outline" size="sm" onclick={load} disabled={loading}>
|
||||
<Button variant="outline" size="sm" onclick={() => void load()} disabled={loading}>
|
||||
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
|
||||
Обновить
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="mb-4 flex flex-wrap items-end gap-2">
|
||||
<label class="text-xs text-muted-foreground">
|
||||
Aliases
|
||||
<input
|
||||
type="text"
|
||||
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
|
||||
bind:value={aliasesInput}
|
||||
placeholder="node-a,node-b"
|
||||
/>
|
||||
</label>
|
||||
<label class="text-xs text-muted-foreground">
|
||||
Refresh (sec)
|
||||
<input
|
||||
type="number"
|
||||
min="0"
|
||||
max="300"
|
||||
class="mt-1 h-9 w-28 rounded-md border bg-background px-2 text-sm"
|
||||
bind:value={refreshInput}
|
||||
/>
|
||||
</label>
|
||||
<Button variant="outline" size="sm" onclick={() => void applyControls()} disabled={loading}>
|
||||
Apply
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{#if partial}
|
||||
<Alert class="mb-4">
|
||||
<AlertTitle>Частичные данные</AlertTitle>
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
<script lang="ts">
|
||||
import { onMount } from 'svelte';
|
||||
import { page } from '$app/state';
|
||||
import { goto } from '$app/navigation';
|
||||
import {
|
||||
liveEventsUrl,
|
||||
type IncidentItem,
|
||||
type IncidentSeverity,
|
||||
type IncidentStatus
|
||||
} from '$lib/api/client.js';
|
||||
import * as Card from '$lib/components/ui/card/index.js';
|
||||
import { Alert, AlertDescription, AlertTitle } from '$lib/components/ui/alert/index.js';
|
||||
import { Badge } from '$lib/components/ui/badge/index.js';
|
||||
import { Button } from '$lib/components/ui/button/index.js';
|
||||
import * as Table from '$lib/components/ui/table/index.js';
|
||||
import RefreshCwIcon from '@lucide/svelte/icons/refresh-cw';
|
||||
|
||||
type LiveStatus = 'healthy' | 'degraded' | 'critical';
|
||||
|
||||
type LiveSnapshot = {
|
||||
type?: string;
|
||||
timestamp?: string;
|
||||
status?: LiveStatus;
|
||||
partial?: boolean;
|
||||
incidents?: IncidentItem[];
|
||||
counts?: {
|
||||
total?: number;
|
||||
critical?: number;
|
||||
warning?: number;
|
||||
info?: number;
|
||||
};
|
||||
aliases_used?: string[];
|
||||
};
|
||||
|
||||
let aliases = $state('');
|
||||
let aliasesInput = $state('');
|
||||
let err = $state<string | null>(null);
|
||||
let connected = $state(false);
|
||||
let reconnectInSec = $state<number | null>(null);
|
||||
let reconnectAttempt = $state(0);
|
||||
let snapshot = $state<LiveSnapshot | null>(null);
|
||||
let lastEventAtMs = $state<number | null>(null);
|
||||
let nowMs = $state(Date.now());
|
||||
let reconnectTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
let staleTimer: ReturnType<typeof setInterval> | null = null;
|
||||
let es: EventSource | null = null;
|
||||
|
||||
function severityVariant(sev: IncidentSeverity): 'default' | 'secondary' | 'destructive' | 'outline' {
|
||||
if (sev === 'critical') return 'destructive';
|
||||
if (sev === 'warning') return 'secondary';
|
||||
return 'outline';
|
||||
}
|
||||
|
||||
function statusVariant(st: LiveStatus | undefined): 'default' | 'secondary' | 'destructive' {
|
||||
if (st === 'critical') return 'destructive';
|
||||
if (st === 'degraded') return 'secondary';
|
||||
return 'default';
|
||||
}
|
||||
|
||||
function normIncident(v: unknown): IncidentItem | null {
|
||||
if (!v || typeof v !== 'object') return null;
|
||||
const x = v as Record<string, unknown>;
|
||||
const id = typeof x.id === 'string' ? x.id : '';
|
||||
if (!id) return null;
|
||||
const severity = (x.severity ?? 'info') as IncidentSeverity;
|
||||
const status = (x.status ?? 'firing') as IncidentStatus;
|
||||
return {
|
||||
id,
|
||||
kind: typeof x.kind === 'string' ? x.kind : '',
|
||||
severity,
|
||||
status,
|
||||
title: typeof x.title === 'string' ? x.title : id,
|
||||
summary: typeof x.summary === 'string' ? x.summary : '',
|
||||
affected_aliases: Array.isArray(x.affected_aliases)
|
||||
? x.affected_aliases.filter((a): a is string => typeof a === 'string')
|
||||
: [],
|
||||
metric_name: typeof x.metric_name === 'string' ? x.metric_name : undefined,
|
||||
metric_value: typeof x.metric_value === 'number' ? x.metric_value : undefined,
|
||||
metric_threshold: typeof x.metric_threshold === 'number' ? x.metric_threshold : undefined,
|
||||
actions: Array.isArray(x.actions)
|
||||
? x.actions
|
||||
.filter((a) => a && typeof a === 'object')
|
||||
.map((a) => a as Record<string, unknown>)
|
||||
.filter((a) => typeof a.label === 'string' && typeof a.href === 'string')
|
||||
.map((a) => ({ label: String(a.label), href: String(a.href) }))
|
||||
: []
|
||||
};
|
||||
}
|
||||
|
||||
function parseSnapshot(raw: unknown): LiveSnapshot | null {
|
||||
if (!raw || typeof raw !== 'object') return null;
|
||||
const x = raw as Record<string, unknown>;
|
||||
const incidents = Array.isArray(x.incidents) ? x.incidents.map(normIncident).filter(Boolean) : [];
|
||||
const countsRaw = x.counts && typeof x.counts === 'object' ? (x.counts as Record<string, unknown>) : {};
|
||||
return {
|
||||
type: typeof x.type === 'string' ? x.type : undefined,
|
||||
timestamp: typeof x.timestamp === 'string' ? x.timestamp : undefined,
|
||||
status:
|
||||
x.status === 'healthy' || x.status === 'degraded' || x.status === 'critical'
|
||||
? x.status
|
||||
: 'healthy',
|
||||
partial: !!x.partial,
|
||||
incidents: incidents as IncidentItem[],
|
||||
counts: {
|
||||
total: typeof countsRaw.total === 'number' ? countsRaw.total : incidents.length,
|
||||
critical: typeof countsRaw.critical === 'number' ? countsRaw.critical : undefined,
|
||||
warning: typeof countsRaw.warning === 'number' ? countsRaw.warning : undefined,
|
||||
info: typeof countsRaw.info === 'number' ? countsRaw.info : undefined
|
||||
},
|
||||
aliases_used: Array.isArray(x.aliases_used)
|
||||
? x.aliases_used.filter((a): a is string => typeof a === 'string')
|
||||
: []
|
||||
};
|
||||
}
|
||||
|
||||
function clearReconnectTimer() {
|
||||
if (!reconnectTimer) return;
|
||||
clearTimeout(reconnectTimer);
|
||||
reconnectTimer = null;
|
||||
}
|
||||
|
||||
function closeStream() {
|
||||
if (es) {
|
||||
es.close();
|
||||
es = null;
|
||||
}
|
||||
connected = false;
|
||||
}
|
||||
|
||||
function scheduleReconnect() {
|
||||
clearReconnectTimer();
|
||||
const delayMs = Math.min(10_000, 1000 * 2 ** Math.min(8, reconnectAttempt));
|
||||
reconnectAttempt += 1;
|
||||
reconnectInSec = Math.ceil(delayMs / 1000);
|
||||
const target = Date.now() + delayMs;
|
||||
reconnectTimer = setTimeout(() => {
|
||||
reconnectInSec = null;
|
||||
connectStream();
|
||||
}, delayMs);
|
||||
const countdown = setInterval(() => {
|
||||
const left = Math.max(0, Math.ceil((target - Date.now()) / 1000));
|
||||
reconnectInSec = left;
|
||||
if (left <= 0) clearInterval(countdown);
|
||||
}, 250);
|
||||
}
|
||||
|
||||
function connectStream() {
|
||||
closeStream();
|
||||
clearReconnectTimer();
|
||||
err = null;
|
||||
const url = liveEventsUrl({ aliases: aliases || undefined });
|
||||
const next = new EventSource(url);
|
||||
next.onopen = () => {
|
||||
connected = true;
|
||||
reconnectAttempt = 0;
|
||||
reconnectInSec = null;
|
||||
};
|
||||
next.onmessage = (ev) => {
|
||||
try {
|
||||
const parsed = parseSnapshot(JSON.parse(ev.data));
|
||||
if (!parsed) return;
|
||||
snapshot = parsed;
|
||||
lastEventAtMs = Date.now();
|
||||
err = null;
|
||||
connected = true;
|
||||
} catch (e) {
|
||||
err = `Невалидный snapshot: ${String(e)}`;
|
||||
}
|
||||
};
|
||||
next.addEventListener('snapshot', (ev) => {
|
||||
try {
|
||||
const msg = ev as MessageEvent<string>;
|
||||
const parsed = parseSnapshot(JSON.parse(msg.data));
|
||||
if (!parsed) return;
|
||||
snapshot = parsed;
|
||||
lastEventAtMs = Date.now();
|
||||
err = null;
|
||||
connected = true;
|
||||
} catch (e) {
|
||||
err = `Невалидный snapshot: ${String(e)}`;
|
||||
}
|
||||
});
|
||||
next.onerror = () => {
|
||||
connected = false;
|
||||
err = 'Поток SSE разорван, переподключение...';
|
||||
next.close();
|
||||
if (es === next) es = null;
|
||||
scheduleReconnect();
|
||||
};
|
||||
es = next;
|
||||
}
|
||||
|
||||
async function applyControls() {
|
||||
const q = new URLSearchParams(page.url.searchParams);
|
||||
const normalizedAliases = aliasesInput.trim();
|
||||
if (normalizedAliases) q.set('aliases', normalizedAliases);
|
||||
else q.delete('aliases');
|
||||
const qs = q.toString();
|
||||
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
|
||||
replaceState: true,
|
||||
noScroll: true,
|
||||
keepFocus: true
|
||||
});
|
||||
}
|
||||
|
||||
let queryKey = $derived(page.url.searchParams.toString());
|
||||
$effect(() => {
|
||||
queryKey;
|
||||
if (typeof window === 'undefined') return;
|
||||
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
|
||||
aliasesInput = aliases;
|
||||
connectStream();
|
||||
});
|
||||
|
||||
let staleSeconds = $derived.by(() =>
|
||||
lastEventAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastEventAtMs) / 1000))
|
||||
);
|
||||
let isStale = $derived.by(() => staleSeconds == null || staleSeconds > 12);
|
||||
|
||||
onMount(() => {
|
||||
staleTimer = setInterval(() => {
|
||||
nowMs = Date.now();
|
||||
}, 1000);
|
||||
return () => {
|
||||
closeStream();
|
||||
clearReconnectTimer();
|
||||
if (staleTimer) clearInterval(staleTimer);
|
||||
};
|
||||
});
|
||||
</script>
|
||||
|
||||
<div class="mb-6 flex flex-wrap items-center justify-between gap-4">
|
||||
<div>
|
||||
<h1 class="text-2xl font-semibold tracking-tight">Live</h1>
|
||||
<p class="text-sm text-muted-foreground">SSE поток инцидентов и статуса флота.</p>
|
||||
</div>
|
||||
<div class="flex items-center gap-2">
|
||||
<Badge variant={connected ? 'default' : 'secondary'}>
|
||||
{connected ? 'connected' : 'disconnected'}
|
||||
</Badge>
|
||||
<Badge variant={isStale ? 'secondary' : 'default'}>
|
||||
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
|
||||
</Badge>
|
||||
<Button variant="outline" size="sm" onclick={() => connectStream()}>
|
||||
<RefreshCwIcon class="mr-1 size-4" />
|
||||
Reconnect
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="mb-4 flex flex-wrap items-end gap-2">
|
||||
<label class="text-xs text-muted-foreground">
|
||||
Aliases
|
||||
<input
|
||||
type="text"
|
||||
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
|
||||
bind:value={aliasesInput}
|
||||
placeholder="node-a,node-b"
|
||||
/>
|
||||
</label>
|
||||
<Button variant="outline" size="sm" onclick={() => void applyControls()}>
|
||||
Apply
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{#if err}
|
||||
<Alert class="mb-4">
|
||||
<AlertTitle>Live stream</AlertTitle>
|
||||
<AlertDescription>
|
||||
{err}
|
||||
{#if reconnectInSec != null}
|
||||
Переподключение через {reconnectInSec}s.
|
||||
{/if}
|
||||
</AlertDescription>
|
||||
</Alert>
|
||||
{/if}
|
||||
|
||||
<div class="mb-4 grid gap-4 sm:grid-cols-3">
|
||||
<Card.Root>
|
||||
<Card.Header class="pb-2">
|
||||
<Card.Description>Общий статус</Card.Description>
|
||||
<Card.Title>
|
||||
<Badge variant={statusVariant(snapshot?.status)}>
|
||||
{snapshot?.status ?? 'healthy'}
|
||||
</Badge>
|
||||
</Card.Title>
|
||||
</Card.Header>
|
||||
</Card.Root>
|
||||
<Card.Root>
|
||||
<Card.Header class="pb-2">
|
||||
<Card.Description>Partial</Card.Description>
|
||||
<Card.Title class="text-2xl tabular-nums">{snapshot?.partial ? 'yes' : 'no'}</Card.Title>
|
||||
</Card.Header>
|
||||
</Card.Root>
|
||||
<Card.Root>
|
||||
<Card.Header class="pb-2">
|
||||
<Card.Description>Timestamp</Card.Description>
|
||||
<Card.Title class="text-sm">
|
||||
{snapshot?.timestamp ? new Date(snapshot.timestamp).toLocaleString() : '—'}
|
||||
</Card.Title>
|
||||
</Card.Header>
|
||||
</Card.Root>
|
||||
</div>
|
||||
|
||||
<Card.Root>
|
||||
<Card.Header>
|
||||
<Card.Title>Последние incidents из snapshot</Card.Title>
|
||||
<Card.Description>Всего: {snapshot?.counts?.total ?? snapshot?.incidents?.length ?? 0}</Card.Description>
|
||||
</Card.Header>
|
||||
<Card.Content class="p-0">
|
||||
<Table.Root>
|
||||
<Table.Header>
|
||||
<Table.Row>
|
||||
<Table.Head>Severity</Table.Head>
|
||||
<Table.Head>Title</Table.Head>
|
||||
<Table.Head>Summary</Table.Head>
|
||||
<Table.Head>Aliases</Table.Head>
|
||||
</Table.Row>
|
||||
</Table.Header>
|
||||
<Table.Body>
|
||||
{#if (snapshot?.incidents?.length ?? 0) === 0}
|
||||
<Table.Row>
|
||||
<Table.Cell colspan={4} class="p-4 text-center text-muted-foreground">
|
||||
Инцидентов нет
|
||||
</Table.Cell>
|
||||
</Table.Row>
|
||||
{:else}
|
||||
{#each snapshot?.incidents ?? [] as item (item.id)}
|
||||
<Table.Row>
|
||||
<Table.Cell>
|
||||
<Badge variant={severityVariant(item.severity)}>{item.severity}</Badge>
|
||||
</Table.Cell>
|
||||
<Table.Cell class="font-medium">{item.title}</Table.Cell>
|
||||
<Table.Cell class="max-w-[380px] text-sm text-muted-foreground">{item.summary}</Table.Cell>
|
||||
<Table.Cell>
|
||||
<div class="flex flex-wrap gap-1">
|
||||
{#each item.affected_aliases ?? [] as alias (alias)}
|
||||
<Badge variant="outline">{alias}</Badge>
|
||||
{/each}
|
||||
</div>
|
||||
</Table.Cell>
|
||||
</Table.Row>
|
||||
{/each}
|
||||
{/if}
|
||||
</Table.Body>
|
||||
</Table.Root>
|
||||
</Card.Content>
|
||||
</Card.Root>
|
||||
@@ -1,9 +1,11 @@
|
||||
<script lang="ts">
|
||||
import { onMount } from 'svelte';
|
||||
import { page } from '$app/state';
|
||||
import { goto } from '$app/navigation';
|
||||
import { fetchAggUsers, ApiError } from '$lib/api/client.js';
|
||||
import type { components } from '$lib/api/aggregate.gen.js';
|
||||
import { formatMiB } from '$lib/format.js';
|
||||
import { Badge } from '$lib/components/ui/badge/index.js';
|
||||
import { Button } from '$lib/components/ui/button/index.js';
|
||||
import * as Table from '$lib/components/ui/table/index.js';
|
||||
import * as Card from '$lib/components/ui/card/index.js';
|
||||
@@ -16,14 +18,39 @@
|
||||
let rows = $state<components['schemas']['UsersRow'][]>([]);
|
||||
let partial = $state(false);
|
||||
let includeLinks = $state(false);
|
||||
let aliases = $state('');
|
||||
let aliasesInput = $state('');
|
||||
let refreshSeconds = $state(30);
|
||||
let refreshInput = $state('30');
|
||||
let lastSuccessAtMs = $state<number | null>(null);
|
||||
let nowMs = $state(Date.now());
|
||||
let pollTimer: ReturnType<typeof setInterval> | null = null;
|
||||
let staleTimer: ReturnType<typeof setInterval> | null = null;
|
||||
|
||||
function parseRefresh(raw: string | null): number {
|
||||
if (raw == null || raw.trim() === '') return 30;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n)) return 30;
|
||||
if (n === 0) return 0;
|
||||
return Math.max(10, Math.min(300, Math.floor(n)));
|
||||
}
|
||||
|
||||
function parse01(raw: string | null, fallback: boolean): boolean {
|
||||
if (raw == null || raw === '') return fallback;
|
||||
return raw === '1';
|
||||
}
|
||||
|
||||
async function load() {
|
||||
loading = true;
|
||||
err = null;
|
||||
try {
|
||||
const usersEnv = await fetchAggUsers({ include_links: includeLinks });
|
||||
const usersEnv = await fetchAggUsers({
|
||||
aliases: aliases || undefined,
|
||||
include_links: includeLinks
|
||||
});
|
||||
partial = !!usersEnv.partial;
|
||||
rows = usersEnv.data ?? [];
|
||||
lastSuccessAtMs = Date.now();
|
||||
} catch (e) {
|
||||
err = e instanceof ApiError ? e.message : String(e);
|
||||
} finally {
|
||||
@@ -31,7 +58,64 @@
|
||||
}
|
||||
}
|
||||
|
||||
onMount(load);
|
||||
function restartPolling() {
|
||||
if (pollTimer) {
|
||||
clearInterval(pollTimer);
|
||||
pollTimer = null;
|
||||
}
|
||||
if (refreshSeconds > 0) {
|
||||
pollTimer = setInterval(() => {
|
||||
void load();
|
||||
}, refreshSeconds * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
async function applyControls() {
|
||||
const q = new URLSearchParams(page.url.searchParams);
|
||||
const normalizedAliases = aliasesInput.trim();
|
||||
const parsedRefresh = parseRefresh(refreshInput);
|
||||
if (normalizedAliases) q.set('aliases', normalizedAliases);
|
||||
else q.delete('aliases');
|
||||
if (parsedRefresh === 30) q.delete('refresh');
|
||||
else q.set('refresh', String(parsedRefresh));
|
||||
q.set('include_links', includeLinks ? '1' : '0');
|
||||
const qs = q.toString();
|
||||
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
|
||||
replaceState: true,
|
||||
noScroll: true,
|
||||
keepFocus: true
|
||||
});
|
||||
}
|
||||
|
||||
let queryKey = $derived(page.url.searchParams.toString());
|
||||
$effect(() => {
|
||||
queryKey;
|
||||
if (typeof window === 'undefined') return;
|
||||
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
|
||||
refreshSeconds = parseRefresh(page.url.searchParams.get('refresh'));
|
||||
includeLinks = parse01(page.url.searchParams.get('include_links'), false);
|
||||
aliasesInput = aliases;
|
||||
refreshInput = String(refreshSeconds);
|
||||
restartPolling();
|
||||
void load();
|
||||
});
|
||||
|
||||
let staleSeconds = $derived.by(() =>
|
||||
lastSuccessAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastSuccessAtMs) / 1000))
|
||||
);
|
||||
let isStale = $derived.by(() =>
|
||||
staleSeconds == null ? true : staleSeconds > (refreshSeconds > 0 ? refreshSeconds * 2 : 60)
|
||||
);
|
||||
|
||||
onMount(() => {
|
||||
staleTimer = setInterval(() => {
|
||||
nowMs = Date.now();
|
||||
}, 1000);
|
||||
return () => {
|
||||
if (pollTimer) clearInterval(pollTimer);
|
||||
if (staleTimer) clearInterval(staleTimer);
|
||||
};
|
||||
});
|
||||
|
||||
function copy(text: string) {
|
||||
const value = String(text ?? '');
|
||||
@@ -76,17 +160,50 @@
|
||||
</p>
|
||||
</div>
|
||||
<div class="flex items-center gap-2">
|
||||
<Badge variant={isStale ? 'secondary' : 'default'}>
|
||||
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
|
||||
</Badge>
|
||||
<label class="flex cursor-pointer items-center gap-2 text-sm">
|
||||
<input type="checkbox" bind:checked={includeLinks} onchange={load} class="rounded border" />
|
||||
<input
|
||||
type="checkbox"
|
||||
bind:checked={includeLinks}
|
||||
onchange={() => void applyControls()}
|
||||
class="rounded border"
|
||||
/>
|
||||
Ссылки tg://proxy
|
||||
</label>
|
||||
<Button variant="outline" size="sm" onclick={load} disabled={loading}>
|
||||
<Button variant="outline" size="sm" onclick={() => void load()} disabled={loading}>
|
||||
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
|
||||
Обновить
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="mb-4 flex flex-wrap items-end gap-2">
|
||||
<label class="text-xs text-muted-foreground">
|
||||
Aliases
|
||||
<input
|
||||
type="text"
|
||||
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
|
||||
bind:value={aliasesInput}
|
||||
placeholder="node-a,node-b"
|
||||
/>
|
||||
</label>
|
||||
<label class="text-xs text-muted-foreground">
|
||||
Refresh (sec)
|
||||
<input
|
||||
type="number"
|
||||
min="0"
|
||||
max="300"
|
||||
class="mt-1 h-9 w-28 rounded-md border bg-background px-2 text-sm"
|
||||
bind:value={refreshInput}
|
||||
/>
|
||||
</label>
|
||||
<Button variant="outline" size="sm" onclick={() => void applyControls()} disabled={loading}>
|
||||
Apply
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{#if partial}
|
||||
<Alert class="mb-4">
|
||||
<AlertTitle>Частичные данные</AlertTitle>
|
||||
|
||||
Reference in New Issue
Block a user