Enhance API and UI for incident management and live updates
Publish telemt-api gateway Docker image / test (push) Successful in 24s
Publish telemt-api gateway Docker image / build-and-push (push) Successful in 1m58s

- Added a new endpoint `/api/agg/incidents` to provide a normalized snapshot of incidents for fleet triage, including severity and recommended actions.
- Implemented live event streaming via `/api/live/events` for real-time updates on fleet status and incidents, enhancing observability.
- Updated the Web UI to include dedicated sections for incidents and live updates, improving user navigation and access to critical information.
- Enhanced API documentation to reflect new endpoints and their functionalities, ensuring clarity for developers and users.
This commit is contained in:
Denozordec
2026-03-30 19:17:29 +07:00
parent af11a49c81
commit 8c8ccce6ee
18 changed files with 1655 additions and 28 deletions
+11 -1
View File
@@ -1,6 +1,6 @@
# telemt-api
HTTP‑шлюз на Go для [Telemt Control API](docs/API.md): один порт, **белый список IP (CIDR)**, маршруты вида `/api/{alias}/…``{base_url}/v1/…`, агрегация нескольких инстансов — [`/api/agg/…`](docs/AGGREGATE.md), метрики Prometheus на `/metrics`. **Web UI** (SvelteKit) встроен в тот же процесс/образ: статика на `/`, API на `/api/…` и `/health`.
HTTP‑шлюз на Go для [Telemt Control API](docs/API.md): один порт, **белый список IP (CIDR)**, маршруты вида `/api/{alias}/…``{base_url}/v1/…`, агрегация нескольких инстансов — [`/api/agg/…`](docs/AGGREGATE.md), live SSE поток — `/api/live/events`, метрики Prometheus на `/metrics`. **Web UI** (SvelteKit) встроен в тот же процесс/образ: статика на `/`, API на `/api/…` и `/health`.
## Быстрый старт (Linux)
@@ -81,6 +81,8 @@ cors_allowed_origins:
| **[docs/API.md](docs/API.md)** | Контракт Telemt Control API (`/v1/…`) |
| **[docs/AGGREGATE.md](docs/AGGREGATE.md)** | Агрегирующие эндпоинты шлюза (`/api/agg/…`), CORS, кэш |
| **[docs/AGGREGATE_OPENAPI.yaml](docs/AGGREGATE_OPENAPI.yaml)** | OpenAPI 3 черновик для `/api/agg/*` (генерация типов для UI) |
| **[docs/OPERATIONS_BASELINE.md](docs/OPERATIONS_BASELINE.md)** | Baseline UX/SLO для панели быстрого реагирования (MTTD/MTTR и критерии успеха) |
| **[docs/INCIDENT_ROLLOUT.md](docs/INCIDENT_ROLLOUT.md)** | Пошаговый rollout incidents/live функций и настройка alert policy |
| **[web/README.md](web/README.md)** | Web UI (SvelteKit): разработка с Vite, `PUBLIC_TELEMT_GATEWAY_URL`, встраивание в образ шлюза |
| **[docs/GEOIP.md](docs/GEOIP.md)** | GeoLite2 City (страна/город) и опционально ASN (номер AS, организация) для IP в `unique-ips` |
@@ -90,6 +92,14 @@ cors_allowed_origins:
go mod tidy && go test ./...
```
## Observability (gateway)
Prometheus метрики доступны на `/metrics`, включая:
- `telemt_gateway_http_in_flight`
- `telemt_gateway_http_requests_total{code,method,alias,endpoint}`
- `telemt_gateway_http_request_duration_seconds{method,alias,endpoint}`
## CI/CD
В репозитории: [.gitea/workflows/docker.yaml](.gitea/workflows/docker.yaml) — тесты Go, сборка и публикация образа в Container Registry Gitea (см. раздел «Обновление и CI/CD» в [docs/GATEWAY_RUN.md](docs/GATEWAY_RUN.md)).
+17
View File
@@ -33,6 +33,7 @@
| GET | `/api/agg/users` | Объединённый список пользователей с `by_server`, суммарным `total_megabytes` и **смерженными лимитами** (см. ниже). |
| GET | `/api/agg/user/{username}` | Один пользователь в том же формате, что элементы `/api/agg/users` (без списка всех). Имя в пути: `[A-Za-z0-9_.-]+`. Ответ **`404`**, если пользователь не найден ни на одном успешном upstream. |
| GET | `/api/agg/fleet-status` | По каждому алиасу: параллельно health + system/info; в `data.servers[]` — статусы подзапросов и тела `health` / `system_info` при успехе. См. [AGGREGATE_OPENAPI.yaml](AGGREGATE_OPENAPI.yaml). |
| GET | `/api/agg/incidents` | Нормализованный snapshot инцидентов для triage-панели: `critical/warning/info`, `affected_aliases`, рекомендуемые `actions` (runbook/deep links), счётчики по severity. |
Все методы — **GET**; действует тот же whitelist, что и для остального API шлюза.
@@ -57,6 +58,18 @@
| `include_links` | `users`, `user/…` | `true` — добавить сгенерированные `tg://proxy` ссылки (берётся первая успешная запись по пользователю). |
| `min_total_megabytes` | `users` | Порог суммарного трафика пользователя в MiB (строго больше 0). |
| `min_total_octets` | `users` | Устаревший вариант порога в октетах (если задан `min_total_megabytes`, он приоритетнее). |
| `aliases` | `incidents` | Список алиасов через запятую; позволяет строить incidents snapshot по выбранной группе нод. |
## Live stream (SSE)
Для оперативного режима NOC доступен поток событий:
- **`GET /api/live/events`** (`text/event-stream`)
- query: `aliases` (опционально, как в `/api/agg/*`)
- событие: `event: snapshot`
- payload: JSON со статусом флота (`healthy/degraded/critical`), `partial`, и массивом `incidents`
Поток рассчитан на UI-клиент с авто-reconnect (на фронте используется экспоненциальный backoff).
## Конфигурация (опционально)
@@ -99,4 +112,8 @@ curl -sS "http://127.0.0.1:8080/api/agg/unique-ips"
curl -sS "http://127.0.0.1:8080/api/agg/users?include_links=false&min_total_megabytes=1"
curl -sS "http://127.0.0.1:8080/api/agg/fleet-status"
curl -sS "http://127.0.0.1:8080/api/agg/user/myuser?aliases=gt1"
curl -sS "http://127.0.0.1:8080/api/agg/incidents?aliases=gt1,gt2"
# SSE поток snapshot-событий (пример с curl)
curl -N "http://127.0.0.1:8080/api/live/events?aliases=gt1,gt2"
```
+59
View File
@@ -106,6 +106,18 @@ paths:
application/json:
schema: { $ref: '#/components/schemas/AggEnvelopeFleetStatus' }
/api/agg/incidents:
get:
summary: Нормализованный snapshot инцидентов по флоту
parameters:
- $ref: '#/components/parameters/aliases'
responses:
'200':
description: OK
content:
application/json:
schema: { $ref: '#/components/schemas/AggEnvelopeIncidents' }
components:
parameters:
aliases:
@@ -173,6 +185,13 @@ components:
properties:
data: { $ref: '#/components/schemas/FleetStatusData' }
AggEnvelopeIncidents:
allOf:
- $ref: '#/components/schemas/AggSuccessBase'
- type: object
properties:
data: { $ref: '#/components/schemas/IncidentsData' }
TrafficRow:
type: object
properties:
@@ -297,3 +316,43 @@ components:
servers_total: { type: integer }
servers_all_ok: { type: integer }
servers_failed: { type: integer }
IncidentAction:
type: object
properties:
label: { type: string }
href: { type: string }
IncidentItem:
type: object
properties:
id: { type: string }
kind: { type: string }
severity:
type: string
enum: [info, warning, critical]
status:
type: string
enum: [firing]
title: { type: string }
summary: { type: string }
affected_aliases:
type: array
items: { type: string }
metric_name: { type: string }
metric_value: { type: number, format: float }
metric_threshold: { type: number, format: float }
actions:
type: array
items: { $ref: '#/components/schemas/IncidentAction' }
IncidentsData:
type: object
properties:
items:
type: array
items: { $ref: '#/components/schemas/IncidentItem' }
total: { type: integer }
critical_total: { type: integer }
warning_total: { type: integer }
info_total: { type: integer }
+73
View File
@@ -0,0 +1,73 @@
# Incident Rollout Playbook
## Scope
This playbook describes staged rollout for:
- `/api/agg/incidents`
- `/api/live/events` (SSE snapshots)
- UI pages `/incidents` and `/live`
- Live polling controls and stale indicators on key pages
## Stage 0 - Baseline Capture (2-3 days)
- Record current MTTD and MTTR from on-call logs.
- Record manual refresh usage on main pages.
- Save top recurring failure patterns (degraded nodes, read-only modes, bad connections spikes).
Outputs:
- baseline MTTD / MTTR
- top 5 incident categories by frequency
## Stage 1 - Shadow Mode (3-5 days)
- Enable incidents and live pages for operators.
- Do not change paging/escalation yet.
- Compare incident feed against existing monitoring and mark false positives.
Targets:
- false positive ratio < 20%
- no increase in upstream load beyond acceptable budget
## Stage 2 - Assisted Triage (1 week)
- Use `/incidents` as primary triage board.
- Require owner + ack for active critical incidents.
- Use runbook links from incident items.
Targets:
- ack coverage for critical incidents >= 90%
- owner coverage for critical incidents >= 90%
## Stage 3 - Policy Tuning (ongoing)
- Adjust thresholds:
- `bad_connections_warn` (default 1000)
- `bad_connections_high` (default 10000)
- Review alert fatigue weekly.
- Promote stable thresholds into documented policy.
## KPI Tracking
- MTTD (minutes): incident first observed -> first ack
- MTTR (minutes): incident first observed -> resolved
- Stale time share: percentage of time live views are stale
- Manual refresh share: manual refresh / total data update actions
## Fast Rollback
If noise or load is excessive:
1. disable auto-refresh by setting `refresh=0` in shared ops links
2. switch operators back to dashboard summary only
3. keep `/api/agg/incidents` for diagnostics while disabling SSE consumers
## Weekly Review Template
- KPI delta (MTTD, MTTR) vs baseline
- top noisy rules
- incidents with missing owner/ack
- policy changes applied this week
+49
View File
@@ -0,0 +1,49 @@
# Operations Baseline (Telemt Panel)
## Purpose
This document defines the baseline operating model and target SLO metrics for the Telemt response panel.
It is used to measure impact of live updates, incidents workflow, and observability improvements.
## Current Baseline (Before Rollout)
- Dashboard refresh mode: mostly manual (`Refresh` buttons on key pages).
- Unified incident queue: not present.
- Alert ownership and acknowledgement flow: not present.
- MTTR tracking: not formalized in product UI.
- MTTD tracking: not formalized in product UI.
- Cross-page filtering consistency: partial, per-page controls only.
## Baseline Risks
- Slow detection when operators do not refresh at the right time.
- High context switching cost between pages during incidents.
- No single place to triage partial/degraded node states.
- Weak evidence trail for post-incident analysis.
## SLO Targets (Phase Goals)
- MTTD reduction: at least 30% versus baseline.
- MTTR reduction: at least 25% versus baseline.
- Manual refresh share during incident windows: under 10%.
- Incident cards with `owner + ack + runbook`: over 90%.
## Measurement Inputs
- Gateway API:
- `/api/agg/fleet-status`
- `/api/agg/summary`
- `/api/agg/incidents` (planned/implemented in this rollout)
- Gateway metrics endpoint:
- `/metrics`
- Frontend telemetry (local panel interaction metrics):
- refresh actions
- ack/resolve actions
- stale/live state durations
## Rollout Validation Checklist
- [ ] Baseline values captured before enabling auto-refresh.
- [ ] Incident flow tested with simulated degraded upstream.
- [ ] Alert noise review completed after first week.
- [ ] MTTD/MTTR comparison published for phase review.
+19
View File
@@ -35,6 +35,11 @@ type Handler struct {
cache map[string]cacheEntry
}
// ResolveAliasesForLive resolves aliases for external endpoints (SSE/live).
func (h *Handler) ResolveAliasesForLive(r *http.Request) ([]string, error) {
return h.resolveAliases(r)
}
// NewHandler builds an aggregate handler; client must use a non-nil Transport (e.g. gateway shared transport).
// Geo may be nil (no GeoLite2 lookups). cacheTTL 0 disables response caching.
func NewHandler(p *config.Parsed, client *http.Client, geo *geoip.Service, cacheTTL time.Duration) *Handler {
@@ -108,6 +113,8 @@ func (h *Handler) dispatch(w http.ResponseWriter, r *http.Request, sub string) {
h.handleUsers(w, r)
case sub == "fleet-status":
h.handleFleetStatus(w, r)
case sub == "incidents":
h.handleIncidents(w, r)
case strings.HasPrefix(sub, "user/"):
username := strings.TrimPrefix(sub, "user/")
if username == "" {
@@ -304,6 +311,18 @@ func (h *Handler) handleUserOne(w http.ResponseWriter, r *http.Request, username
writeAggOK(w, partial, row)
}
func (h *Handler) handleIncidents(w http.ResponseWriter, r *http.Request) {
aliases, err := h.resolveAliases(r)
if err != nil {
writeBadRequest(w, err)
return
}
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
defer cancel()
data, partial := BuildIncidents(ctx, h, aliases)
writeAggOK(w, partial, data)
}
func writeBadRequest(w http.ResponseWriter, err error) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(http.StatusBadRequest)
+154
View File
@@ -0,0 +1,154 @@
package aggregate
import (
"context"
"fmt"
"sort"
"strings"
"time"
)
type statsSummaryLite struct {
ConnectionsBadTotal uint64 `json:"connections_bad_total"`
ConnectionsTotal uint64 `json:"connections_total"`
}
func BuildIncidents(ctx context.Context, h *Handler, aliases []string) (IncidentsData, bool) {
out := IncidentsData{Items: make([]IncidentItem, 0)}
partial := false
fleet := FetchFleetStatus(ctx, h.Client, h.Parsed, aliases)
if fleet.ServersFailed > 0 {
partial = true
}
for _, s := range fleet.Servers {
if !s.OK {
severity := SeverityWarning
if !s.HealthOK && !s.SystemInfoOK {
severity = SeverityCritical
}
out.Items = append(out.Items, IncidentItem{
ID: fmt.Sprintf("node_degraded:%s", s.Alias),
Kind: "node_degraded",
Severity: severity,
Status: IncidentStatusFiring,
Title: fmt.Sprintf("Нода %s degraded", s.Alias),
Summary: "health/system_info не прошли полностью",
AffectedAliases: []string{s.Alias},
Actions: []IncidentAction{
{Label: "Открыть ноду", Href: "/servers/" + s.Alias},
{Label: "Runtime", Href: "/servers/" + s.Alias + "/runtime"},
},
})
}
if s.Health != nil && s.Health.ReadOnly {
out.Items = append(out.Items, IncidentItem{
ID: fmt.Sprintf("node_read_only:%s", s.Alias),
Kind: "node_read_only",
Severity: SeverityWarning,
Status: IncidentStatusFiring,
Title: fmt.Sprintf("Нода %s в read_only", s.Alias),
Summary: "API ноды не принимает mutating операции",
AffectedAliases: []string{s.Alias},
Actions: []IncidentAction{
{Label: "Users", Href: "/servers/" + s.Alias + "/users"},
{Label: "Security", Href: "/servers/" + s.Alias + "/security"},
},
})
}
}
for _, alias := range aliases {
data, meta := FetchTelemtGET[statsSummaryLite](ctx, h.Client, h.Parsed, alias, "stats/summary")
if !meta.OK {
partial = true
continue
}
if data.ConnectionsBadTotal >= 10000 {
out.Items = append(out.Items, IncidentItem{
ID: fmt.Sprintf("bad_connections_high:%s", alias),
Kind: "bad_connections_high",
Severity: SeverityCritical,
Status: IncidentStatusFiring,
Title: fmt.Sprintf("Высокий bad connections на %s", alias),
Summary: "Резкий рост ошибок клиентских соединений",
AffectedAliases: []string{alias},
MetricName: "connections_bad_total",
MetricValue: float64(data.ConnectionsBadTotal),
MetricThreshold: 10000,
Actions: []IncidentAction{
{Label: "Node dashboard", Href: "/servers/" + alias},
},
})
} else if data.ConnectionsBadTotal >= 1000 {
out.Items = append(out.Items, IncidentItem{
ID: fmt.Sprintf("bad_connections_warn:%s", alias),
Kind: "bad_connections_warn",
Severity: SeverityWarning,
Status: IncidentStatusFiring,
Title: fmt.Sprintf("Рост bad connections на %s", alias),
Summary: "Наблюдается рост ошибок клиентских соединений",
AffectedAliases: []string{alias},
MetricName: "connections_bad_total",
MetricValue: float64(data.ConnectionsBadTotal),
MetricThreshold: 1000,
Actions: []IncidentAction{
{Label: "Node dashboard", Href: "/servers/" + alias},
},
})
}
}
sort.Slice(out.Items, func(i, j int) bool {
if severityWeight(out.Items[i].Severity) != severityWeight(out.Items[j].Severity) {
return severityWeight(out.Items[i].Severity) > severityWeight(out.Items[j].Severity)
}
return strings.Compare(out.Items[i].ID, out.Items[j].ID) < 0
})
out.Total = len(out.Items)
for _, it := range out.Items {
switch it.Severity {
case SeverityCritical:
out.CriticalTotal++
case SeverityWarning:
out.WarningTotal++
default:
out.InfoTotal++
}
}
return out, partial
}
func severityWeight(s IncidentSeverity) int {
switch s {
case SeverityCritical:
return 3
case SeverityWarning:
return 2
case SeverityInfo:
return 1
default:
return 0
}
}
func BuildLiveEnvelope(ctx context.Context, h *Handler, aliases []string) map[string]any {
inc, partial := BuildIncidents(ctx, h, aliases)
status := "healthy"
if inc.CriticalTotal > 0 {
status = "critical"
} else if inc.WarningTotal > 0 {
status = "degraded"
}
return map[string]any{
"type": "live_snapshot",
"timestamp": time.Now().UTC().Format(time.RFC3339Nano),
"status": status,
"partial": partial,
"incidents": inc.Items,
"counts": map[string]int{"total": inc.Total, "critical": inc.CriticalTotal, "warning": inc.WarningTotal, "info": inc.InfoTotal},
"aliases_used": aliases,
}
}
+46
View File
@@ -149,6 +149,52 @@ type SummaryData struct {
TopUsersByUniqueIPs []TopUserByUniqueIPs `json:"top_users_by_unique_ips"`
}
// IncidentSeverity is normalized severity used by aggregate incidents.
type IncidentSeverity string
const (
SeverityInfo IncidentSeverity = "info"
SeverityWarning IncidentSeverity = "warning"
SeverityCritical IncidentSeverity = "critical"
)
// IncidentStatus is current calculated incident state.
type IncidentStatus string
const (
IncidentStatusFiring IncidentStatus = "firing"
)
// IncidentAction points to a UI route with details/runbook context.
type IncidentAction struct {
Label string `json:"label"`
Href string `json:"href"`
}
// IncidentItem is one normalized fleet incident.
type IncidentItem struct {
ID string `json:"id"`
Kind string `json:"kind"`
Severity IncidentSeverity `json:"severity"`
Status IncidentStatus `json:"status"`
Title string `json:"title"`
Summary string `json:"summary"`
AffectedAliases []string `json:"affected_aliases,omitempty"`
MetricName string `json:"metric_name,omitempty"`
MetricValue float64 `json:"metric_value,omitempty"`
MetricThreshold float64 `json:"metric_threshold,omitempty"`
Actions []IncidentAction `json:"actions,omitempty"`
}
// IncidentsData is aggregate response payload for /api/agg/incidents.
type IncidentsData struct {
Items []IncidentItem `json:"items"`
Total int `json:"total"`
CriticalTotal int `json:"critical_total"`
WarningTotal int `json:"warning_total"`
InfoTotal int `json:"info_total"`
}
// TopUser by summed traffic across servers for one username (мегабайты).
type TopUser struct {
Username string `json:"username"`
+82 -1
View File
@@ -3,6 +3,7 @@ package server
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"net"
"net/http"
@@ -181,8 +182,9 @@ func (g *Gateway) withMetrics(next http.Handler) http.Handler {
httpInFlight.Inc()
start := time.Now()
alias := routeAlias(r.URL.Path)
endpoint := routeEndpoint(r.URL.Path)
lw := &statusWriter{ResponseWriter: w, status: http.StatusOK}
defer observeRequest(r.Method, alias, lw.status, start)
defer observeRequest(r.Method, alias, endpoint, lw.status, start)
next.ServeHTTP(lw, r)
})
}
@@ -206,6 +208,33 @@ func routeAlias(path string) string {
return rest[:i]
}
func routeEndpoint(path string) string {
if path == "" || path == "/" {
return "ui_root"
}
if path == "/health" || path == "/metrics" {
return strings.TrimPrefix(path, "/")
}
if strings.HasPrefix(path, "/api/agg/") {
return strings.TrimPrefix(path, "/api/agg/")
}
if path == "/api/agg" {
return "agg"
}
if strings.HasPrefix(path, "/api/live/events") {
return "live_events"
}
if strings.HasPrefix(path, "/api/") {
rest := strings.TrimPrefix(path, "/api/")
i := strings.IndexByte(rest, '/')
if i < 0 {
return "proxy_root"
}
return "proxy_" + rest[i+1:]
}
return "ui"
}
func (g *Gateway) serve(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/health":
@@ -233,6 +262,10 @@ func (g *Gateway) serve(w http.ResponseWriter, r *http.Request) {
g.agg.ServeHTTP(w, r)
return
}
if r.URL.Path == "/api/live/events" {
g.serveLiveEvents(w, r)
return
}
if !strings.HasPrefix(r.URL.Path, prefix) {
g.webUI.ServeHTTP(w, r)
return
@@ -265,6 +298,54 @@ func (g *Gateway) serve(w http.ResponseWriter, r *http.Request) {
rp.ServeHTTP(w, r)
}
func (g *Gateway) serveLiveEvents(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
flusher, ok := w.(http.Flusher)
if !ok {
http.Error(w, "stream unsupported", http.StatusInternalServerError)
return
}
aliases, err := g.agg.ResolveAliasesForLive(r)
if err != nil {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{
"ok": false,
"error": map[string]string{"code": "bad_request", "message": err.Error()},
})
return
}
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.WriteHeader(http.StatusOK)
ticker := time.NewTicker(5 * time.Second)
defer ticker.Stop()
push := func() {
ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
defer cancel()
payload := aggregate.BuildLiveEnvelope(ctx, g.agg, aliases)
b, _ := json.Marshal(payload)
_, _ = fmt.Fprintf(w, "event: snapshot\n")
_, _ = fmt.Fprintf(w, "data: %s\n\n", string(b))
flusher.Flush()
}
push()
for {
select {
case <-r.Context().Done():
return
case <-ticker.C:
push()
}
}
}
type statusWriter struct {
http.ResponseWriter
status int
+6 -6
View File
@@ -15,17 +15,17 @@ var (
})
httpRequests = promauto.NewCounterVec(prometheus.CounterOpts{
Name: "telemt_gateway_http_requests_total",
Help: "HTTP requests by status, method, alias.",
}, []string{"code", "method", "alias"})
Help: "HTTP requests by status, method, alias, endpoint.",
}, []string{"code", "method", "alias", "endpoint"})
httpDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "telemt_gateway_http_request_duration_seconds",
Help: "Request duration in seconds.",
Buckets: prometheus.DefBuckets,
}, []string{"method", "alias"})
}, []string{"method", "alias", "endpoint"})
)
func observeRequest(method, alias string, status int, started time.Time) {
func observeRequest(method, alias, endpoint string, status int, started time.Time) {
httpInFlight.Dec()
httpRequests.WithLabelValues(strconv.Itoa(status), method, alias).Inc()
httpDuration.WithLabelValues(method, alias).Observe(time.Since(started).Seconds())
httpRequests.WithLabelValues(strconv.Itoa(status), method, alias, endpoint).Inc()
httpDuration.WithLabelValues(method, alias, endpoint).Observe(time.Since(started).Seconds())
}
+19
View File
@@ -2,6 +2,14 @@
SvelteKit + shadcn-svelte. В **production** статика собирается и **встраивается в образ шлюза** ([Dockerfile](../Dockerfile) в корне репозитория): панель и API на **одном порту** (например `http://127.0.0.1:8080/` — UI, `/api/…` — шлюз).
## Основные разделы панели
- `/` — обзор флота (KPI, активные IP, сводка по нодам)
- `/users`, `/users/[username]` — пользователи и детали
- `/ips` — unique IP + GeoIP карта
- `/incidents` — triage-интерфейс инцидентов (`ack/resolved/owner/note` в localStorage)
- `/live` — live snapshot (SSE, авто-reconnect)
## Переменная `PUBLIC_TELEMT_GATEWAY_URL`
| Значение | Когда |
@@ -42,6 +50,17 @@ npm run gen:api
Источник: [../docs/AGGREGATE_OPENAPI.yaml](../docs/AGGREGATE_OPENAPI.yaml).
## URL-driven controls (операторский режим)
На ключевых страницах используются query-параметры:
- `aliases=node-a,node-b` — фильтр по нодам
- `refresh=0|10..300` — auto-refresh (0 выключает polling)
- `include_links=0|1` — для `/users`
- `geo=0|1` — для `/ips`
Live-страница `/live` работает через SSE endpoint шлюза: `/api/live/events`.
## Ограничения
- Секреты upstream к Telemt задаются на шлюзе (`authorization_env`), не в браузере.
+48
View File
@@ -20,6 +20,37 @@ export type AggEnvelope<T> = {
data: T;
};
export type IncidentSeverity = 'info' | 'warning' | 'critical';
export type IncidentStatus = 'firing';
export type IncidentAction = {
label: string;
href: string;
};
export type IncidentItem = {
id: string;
kind: string;
severity: IncidentSeverity;
status: IncidentStatus;
title: string;
summary: string;
affected_aliases?: string[];
metric_name?: string;
metric_value?: number;
metric_threshold?: number;
actions?: IncidentAction[];
};
export type IncidentsData = {
items: IncidentItem[];
total: number;
critical_total: number;
warning_total: number;
info_total: number;
};
export class ApiError extends Error {
constructor(
message: string,
@@ -120,6 +151,23 @@ export async function fetchAggUser(
return body as AggEnvelope<components['schemas']['UsersRow']>;
}
export async function fetchAggIncidents(params?: { aliases?: string }): Promise<AggEnvelope<IncidentsData>> {
const q = new URLSearchParams();
if (params?.aliases) q.set('aliases', params.aliases);
const url = `${gatewayBase()}/api/agg/incidents${q.toString() ? `?${q}` : ''}`;
const res = await fetch(url);
const body = (await parseJson(res)) as Record<string, unknown> | null;
if (!res.ok) throw new ApiError(`incidents HTTP ${res.status}`, res.status, body);
if (!body || body.ok !== true) throw new ApiError('incidents: ok !== true', res.status, body);
return body as AggEnvelope<IncidentsData>;
}
export function liveEventsUrl(params?: { aliases?: string }): string {
const q = new URLSearchParams();
if (params?.aliases) q.set('aliases', params.aliases);
return `${gatewayBase()}/api/live/events${q.toString() ? `?${q}` : ''}`;
}
/** Путь к upstream без префикса /v1 — шлюз сам добавляет path_prefix. */
function apiUrl(alias: string, path: string): string {
const p = path.replace(/^\/+/, '');
+22
View File
@@ -9,6 +9,8 @@
import NetworkIcon from '@lucide/svelte/icons/network';
import SettingsIcon from '@lucide/svelte/icons/settings';
import RefreshCwIcon from '@lucide/svelte/icons/refresh-cw';
import SirenIcon from '@lucide/svelte/icons/siren';
import RadioIcon from '@lucide/svelte/icons/radio';
import * as Sidebar from '$lib/components/ui/sidebar/index.js';
let {
@@ -76,6 +78,26 @@
{/snippet}
</Sidebar.MenuButton>
</Sidebar.MenuItem>
<Sidebar.MenuItem>
<Sidebar.MenuButton isActive={active('/incidents')} tooltipContent="Incidents">
{#snippet child({ props })}
<a href="/incidents" {...props}>
<SirenIcon />
<span>Инциденты</span>
</a>
{/snippet}
</Sidebar.MenuButton>
</Sidebar.MenuItem>
<Sidebar.MenuItem>
<Sidebar.MenuButton isActive={active('/live')} tooltipContent="Live">
{#snippet child({ props })}
<a href="/live" {...props}>
<RadioIcon />
<span>Live</span>
</a>
{/snippet}
</Sidebar.MenuButton>
</Sidebar.MenuItem>
</Sidebar.Menu>
</Sidebar.Group>
{#if serverAliases.length > 0}
+114 -10
View File
@@ -1,5 +1,7 @@
<script lang="ts">
import { onMount } from 'svelte';
import { page } from '$app/state';
import { goto } from '$app/navigation';
import {
ApiError,
fetchAggFleetStatus,
@@ -73,6 +75,14 @@
let uniqueIps = $state<components['schemas']['UniqueIPsRow'][] | null>(null);
let generatedAt = $state<string | null>(null);
let partial = $state(false);
let aliases = $state('');
let aliasesInput = $state('');
let refreshSeconds = $state(30);
let refreshInput = $state('30');
let lastSuccessAtMs = $state<number | null>(null);
let nowMs = $state(Date.now());
let pollTimer: ReturnType<typeof setInterval> | null = null;
let staleTimer: ReturnType<typeof setInterval> | null = null;
let nodeStats = $state<
Record<
string,
@@ -85,24 +95,25 @@
err = null;
try {
const [s, f, u] = await Promise.all([
fetchAggSummary({ top_n: 15 }),
fetchAggFleetStatus(),
fetchAggUniqueIps({ geo: true })
fetchAggSummary({ top_n: 15, aliases: aliases || undefined }),
fetchAggFleetStatus({ aliases: aliases || undefined }),
fetchAggUniqueIps({ aliases: aliases || undefined, geo: true })
]);
generatedAt = s.generated_at;
partial = !!(s.partial || f.partial || u.partial);
summary = s.data;
fleet = f.data ?? null;
uniqueIps = u.data ?? null;
lastSuccessAtMs = Date.now();
// Как бейдж «OK» в таблице: только health + system/info (без x.ok — в JSON поле ok опционально).
const aliases = (fleet?.servers ?? [])
const okAliases = (fleet?.servers ?? [])
.filter((x) => x.alias && x.health_ok && x.system_info_ok)
.map((x) => x.alias as string);
const next: typeof nodeStats = {};
const concurrency = 4;
for (let i = 0; i < aliases.length; i += concurrency) {
const batch = aliases.slice(i, i + concurrency);
for (let i = 0; i < okAliases.length; i += concurrency) {
const batch = okAliases.slice(i, i + concurrency);
await Promise.all(
batch.map(async (alias) => {
try {
@@ -130,7 +141,70 @@
}
}
onMount(load);
function parseRefresh(raw: string | null): number {
if (raw == null || raw.trim() === '') return 30;
const n = Number(raw);
if (!Number.isFinite(n)) return 30;
if (n === 0) return 0;
return Math.max(10, Math.min(300, Math.floor(n)));
}
function restartPolling() {
if (pollTimer) {
clearInterval(pollTimer);
pollTimer = null;
}
if (refreshSeconds > 0) {
pollTimer = setInterval(() => {
void load();
}, refreshSeconds * 1000);
}
}
async function applyControls() {
const q = new URLSearchParams(page.url.searchParams);
const normalizedAliases = aliasesInput.trim();
const parsedRefresh = parseRefresh(refreshInput);
if (normalizedAliases) q.set('aliases', normalizedAliases);
else q.delete('aliases');
if (parsedRefresh === 30) q.delete('refresh');
else q.set('refresh', String(parsedRefresh));
const qs = q.toString();
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
replaceState: true,
noScroll: true,
keepFocus: true
});
}
let queryKey = $derived(page.url.searchParams.toString());
$effect(() => {
queryKey;
if (typeof window === 'undefined') return;
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
refreshSeconds = parseRefresh(page.url.searchParams.get('refresh'));
aliasesInput = aliases;
refreshInput = String(refreshSeconds);
restartPolling();
void load();
});
let staleSeconds = $derived.by(() =>
lastSuccessAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastSuccessAtMs) / 1000))
);
let isStale = $derived.by(() =>
staleSeconds == null ? true : staleSeconds > (refreshSeconds > 0 ? refreshSeconds * 2 : 60)
);
onMount(() => {
staleTimer = setInterval(() => {
nowMs = Date.now();
}, 1000);
return () => {
if (pollTimer) clearInterval(pollTimer);
if (staleTimer) clearInterval(staleTimer);
};
});
/** Число из ответа Telemt (поле может отсутствовать или прийти строкой). */
function numStat(v: unknown, fallback = 0): number {
@@ -284,9 +358,39 @@
</p>
{/if}
</div>
<Button variant="outline" size="sm" onclick={load} disabled={loading}>
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
Обновить
<div class="flex items-center gap-2">
<Badge variant={isStale ? 'secondary' : 'default'}>
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
</Badge>
<Button variant="outline" size="sm" onclick={() => void load()} disabled={loading}>
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
Обновить
</Button>
</div>
</div>
<div class="mb-4 flex flex-wrap items-end gap-2">
<label class="text-xs text-muted-foreground">
Aliases
<input
type="text"
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
bind:value={aliasesInput}
placeholder="node-a,node-b"
/>
</label>
<label class="text-xs text-muted-foreground">
Refresh (sec)
<input
type="number"
min="0"
max="300"
class="mt-1 h-9 w-28 rounded-md border bg-background px-2 text-sm"
bind:value={refreshInput}
/>
</label>
<Button variant="outline" size="sm" onclick={() => void applyControls()} disabled={loading}>
Apply
</Button>
</div>
+351
View File
@@ -0,0 +1,351 @@
<script lang="ts">
import { onMount } from 'svelte';
import { page } from '$app/state';
import { goto } from '$app/navigation';
import {
ApiError,
fetchAggIncidents,
type IncidentItem,
type IncidentsData
} from '$lib/api/client.js';
import * as Card from '$lib/components/ui/card/index.js';
import * as Table from '$lib/components/ui/table/index.js';
import { Alert, AlertDescription, AlertTitle } from '$lib/components/ui/alert/index.js';
import { Badge } from '$lib/components/ui/badge/index.js';
import { Button } from '$lib/components/ui/button/index.js';
import RefreshCwIcon from '@lucide/svelte/icons/refresh-cw';
type TriageState = {
ack?: boolean;
resolved?: boolean;
owner?: string;
note?: string;
};
const TRIAGE_KEY = 'telemt.incidents.triage.v1';
let loading = $state(true);
let err = $state<string | null>(null);
let partial = $state(false);
let generatedAt = $state<string | null>(null);
let data = $state<IncidentsData | null>(null);
let aliases = $state('');
let aliasesInput = $state('');
let refreshSeconds = $state(30);
let refreshInput = $state('30');
let lastSuccessAtMs = $state<number | null>(null);
let nowMs = $state(Date.now());
let pollTimer: ReturnType<typeof setInterval> | null = null;
let staleTimer: ReturnType<typeof setInterval> | null = null;
let triageById = $state<Record<string, TriageState>>({});
function parseRefresh(raw: string | null): number {
if (raw == null || raw.trim() === '') return 30;
const n = Number(raw);
if (!Number.isFinite(n)) return 30;
if (n === 0) return 0;
return Math.max(10, Math.min(300, Math.floor(n)));
}
function triageFor(id: string): TriageState {
return triageById[id] ?? {};
}
function saveTriage() {
if (typeof localStorage === 'undefined') return;
localStorage.setItem(TRIAGE_KEY, JSON.stringify(triageById));
}
function loadTriage() {
if (typeof localStorage === 'undefined') return;
try {
const raw = localStorage.getItem(TRIAGE_KEY);
if (!raw) return;
const parsed = JSON.parse(raw) as Record<string, TriageState>;
if (parsed && typeof parsed === 'object') triageById = parsed;
} catch {
triageById = {};
}
}
function patchTriage(id: string, patch: Partial<TriageState>) {
triageById = {
...triageById,
[id]: {
...triageFor(id),
...patch
}
};
saveTriage();
}
function toggleAck(id: string) {
const t = triageFor(id);
patchTriage(id, { ack: !t.ack });
}
function toggleResolved(id: string) {
const t = triageFor(id);
patchTriage(id, { resolved: !t.resolved });
}
function severityVariant(sev: IncidentItem['severity']): 'default' | 'secondary' | 'destructive' | 'outline' {
if (sev === 'critical') return 'destructive';
if (sev === 'warning') return 'secondary';
return 'outline';
}
function restartPolling() {
if (pollTimer) {
clearInterval(pollTimer);
pollTimer = null;
}
if (refreshSeconds > 0) {
pollTimer = setInterval(() => {
void load();
}, refreshSeconds * 1000);
}
}
async function load() {
loading = true;
err = null;
try {
const env = await fetchAggIncidents({ aliases: aliases || undefined });
data = env.data ?? null;
partial = !!env.partial;
generatedAt = env.generated_at;
lastSuccessAtMs = Date.now();
} catch (e) {
err = e instanceof ApiError ? e.message : String(e);
} finally {
loading = false;
}
}
async function applyControls() {
const q = new URLSearchParams(page.url.searchParams);
const normalizedAliases = aliasesInput.trim();
const parsedRefresh = parseRefresh(refreshInput);
if (normalizedAliases) q.set('aliases', normalizedAliases);
else q.delete('aliases');
if (parsedRefresh === 30) q.delete('refresh');
else q.set('refresh', String(parsedRefresh));
const qs = q.toString();
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
replaceState: true,
noScroll: true,
keepFocus: true
});
}
let queryKey = $derived(page.url.searchParams.toString());
$effect(() => {
queryKey;
if (typeof window === 'undefined') return;
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
refreshSeconds = parseRefresh(page.url.searchParams.get('refresh'));
aliasesInput = aliases;
refreshInput = String(refreshSeconds);
restartPolling();
void load();
});
let staleSeconds = $derived.by(() =>
lastSuccessAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastSuccessAtMs) / 1000))
);
let isStale = $derived.by(() =>
staleSeconds == null ? true : staleSeconds > (refreshSeconds > 0 ? refreshSeconds * 2 : 60)
);
onMount(() => {
loadTriage();
staleTimer = setInterval(() => {
nowMs = Date.now();
}, 1000);
return () => {
if (pollTimer) clearInterval(pollTimer);
if (staleTimer) clearInterval(staleTimer);
};
});
</script>
<div class="mb-6 flex flex-wrap items-center justify-between gap-4">
<div>
<h1 class="text-2xl font-semibold tracking-tight">Инциденты</h1>
<p class="text-sm text-muted-foreground">Сводка и triage по всему флоту.</p>
{#if generatedAt}
<p class="mt-1 text-xs text-muted-foreground">Снимок: {new Date(generatedAt).toLocaleString()}</p>
{/if}
</div>
<div class="flex items-center gap-2">
<Badge variant={isStale ? 'secondary' : 'default'}>
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
</Badge>
<Button variant="outline" size="sm" onclick={() => void load()} disabled={loading}>
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
Обновить
</Button>
</div>
</div>
<div class="mb-4 flex flex-wrap items-end gap-2">
<label class="text-xs text-muted-foreground">
Aliases
<input
type="text"
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
bind:value={aliasesInput}
placeholder="node-a,node-b"
/>
</label>
<label class="text-xs text-muted-foreground">
Refresh (sec)
<input
type="number"
min="0"
max="300"
class="mt-1 h-9 w-28 rounded-md border bg-background px-2 text-sm"
bind:value={refreshInput}
/>
</label>
<Button variant="outline" size="sm" onclick={() => void applyControls()} disabled={loading}>
Apply
</Button>
</div>
{#if partial}
<Alert class="mb-4">
<AlertTitle>Частичные данные</AlertTitle>
<AlertDescription>Не все upstream ответили успешно.</AlertDescription>
</Alert>
{/if}
{#if err}
<Alert variant="destructive">
<AlertTitle>Ошибка</AlertTitle>
<AlertDescription>{err}</AlertDescription>
</Alert>
{:else}
<div class="mb-4 grid gap-4 sm:grid-cols-3">
<Card.Root>
<Card.Header class="pb-2">
<Card.Description>Critical</Card.Description>
<Card.Title class="text-2xl text-red-500 tabular-nums">{data?.critical_total ?? 0}</Card.Title>
</Card.Header>
</Card.Root>
<Card.Root>
<Card.Header class="pb-2">
<Card.Description>Warning</Card.Description>
<Card.Title class="text-2xl text-amber-500 tabular-nums">{data?.warning_total ?? 0}</Card.Title>
</Card.Header>
</Card.Root>
<Card.Root>
<Card.Header class="pb-2">
<Card.Description>Info</Card.Description>
<Card.Title class="text-2xl tabular-nums">{data?.info_total ?? 0}</Card.Title>
</Card.Header>
</Card.Root>
</div>
<Card.Root>
<Card.Content class="p-0">
<Table.Root>
<Table.Header>
<Table.Row>
<Table.Head>Severity</Table.Head>
<Table.Head>Инцидент</Table.Head>
<Table.Head>Серверы</Table.Head>
<Table.Head>Runbook</Table.Head>
<Table.Head>Triage</Table.Head>
</Table.Row>
</Table.Header>
<Table.Body>
{#if (data?.items.length ?? 0) === 0}
<Table.Row>
<Table.Cell colspan={5} class="p-4 text-center text-muted-foreground">
Инцидентов нет
</Table.Cell>
</Table.Row>
{:else}
{#each data?.items ?? [] as item (item.id)}
<Table.Row>
<Table.Cell>
<Badge variant={severityVariant(item.severity)}>{item.severity}</Badge>
</Table.Cell>
<Table.Cell class="max-w-[360px]">
<div class="font-medium">{item.title}</div>
<div class="text-xs text-muted-foreground">{item.summary}</div>
{#if item.metric_name}
<div class="mt-1 text-xs text-muted-foreground">
{item.metric_name}: {item.metric_value} / {item.metric_threshold}
</div>
{/if}
</Table.Cell>
<Table.Cell>
<div class="flex flex-wrap gap-1">
{#each item.affected_aliases ?? [] as alias (alias)}
<Badge variant="outline">{alias}</Badge>
{/each}
</div>
</Table.Cell>
<Table.Cell class="max-w-[240px]">
<div class="flex flex-wrap gap-1">
{#each item.actions ?? [] as action (action.label + action.href)}
<a href={action.href} class="text-primary hover:underline text-xs">
{action.label}
</a>
{/each}
</div>
</Table.Cell>
<Table.Cell class="min-w-[280px]">
{@const triage = triageFor(item.id)}
<div class="mb-2 flex flex-wrap gap-1">
<Button
size="xs"
variant={triage.ack ? 'default' : 'outline'}
onclick={() => toggleAck(item.id)}
>
{triage.ack ? 'unack' : 'ack'}
</Button>
<Button
size="xs"
variant={triage.resolved ? 'default' : 'outline'}
onclick={() => toggleResolved(item.id)}
>
{triage.resolved ? 'unresolve' : 'resolve'}
</Button>
</div>
<div class="mb-1">
<input
type="text"
class="h-8 w-full rounded-md border bg-background px-2 text-xs"
placeholder="owner"
value={triage.owner ?? ''}
oninput={(e) =>
patchTriage(item.id, {
owner: (e.currentTarget as HTMLInputElement).value
})}
/>
</div>
<div>
<input
type="text"
class="h-8 w-full rounded-md border bg-background px-2 text-xs"
placeholder="note"
value={triage.note ?? ''}
oninput={(e) =>
patchTriage(item.id, {
note: (e.currentTarget as HTMLInputElement).value
})}
/>
</div>
</Table.Cell>
</Table.Row>
{/each}
{/if}
</Table.Body>
</Table.Root>
</Card.Content>
</Card.Root>
{/if}
+116 -6
View File
@@ -1,5 +1,7 @@
<script lang="ts">
import { onMount, tick } from 'svelte';
import { page } from '$app/state';
import { goto } from '$app/navigation';
import { fetchAggUniqueIps, ApiError } from '$lib/api/client.js';
import type { components } from '$lib/api/aggregate.gen.js';
import type * as Leaflet from 'leaflet';
@@ -17,6 +19,14 @@
let rows = $state<components['schemas']['UniqueIPsRow'][]>([]);
let partial = $state(false);
let geo = $state(true);
let aliases = $state('');
let aliasesInput = $state('');
let refreshSeconds = $state(30);
let refreshInput = $state('30');
let lastSuccessAtMs = $state<number | null>(null);
let nowMs = $state(Date.now());
let pollTimer: ReturnType<typeof setInterval> | null = null;
let staleTimer: ReturnType<typeof setInterval> | null = null;
let mapEl = $state<HTMLDivElement | null>(null);
let leaflet: typeof import('leaflet') | null = null;
@@ -273,9 +283,10 @@
// Ждём, пока bind:this завершится и контейнеру карты будет доступен размер.
await tick();
initMap();
const env = await fetchAggUniqueIps({ geo });
const env = await fetchAggUniqueIps({ aliases: aliases || undefined, geo });
partial = !!env.partial;
rows = env.data ?? [];
lastSuccessAtMs = Date.now();
} catch (e) {
err = e instanceof ApiError ? e.message : String(e);
} finally {
@@ -288,11 +299,77 @@
}
}
onMount(() => {
// Инициализация Leaflet и загрузка данных происходит в `load()`.
// Это избегает проблем с SSR (если он включён).
function parseRefresh(raw: string | null): number {
if (raw == null || raw.trim() === '') return 30;
const n = Number(raw);
if (!Number.isFinite(n)) return 30;
if (n === 0) return 0;
return Math.max(10, Math.min(300, Math.floor(n)));
}
function parse01(raw: string | null, fallback: boolean): boolean {
if (raw == null || raw === '') return fallback;
return raw === '1';
}
function restartPolling() {
if (pollTimer) {
clearInterval(pollTimer);
pollTimer = null;
}
if (refreshSeconds > 0) {
pollTimer = setInterval(() => {
void load();
}, refreshSeconds * 1000);
}
}
async function applyControls() {
const q = new URLSearchParams(page.url.searchParams);
const normalizedAliases = aliasesInput.trim();
const parsedRefresh = parseRefresh(refreshInput);
if (normalizedAliases) q.set('aliases', normalizedAliases);
else q.delete('aliases');
if (parsedRefresh === 30) q.delete('refresh');
else q.set('refresh', String(parsedRefresh));
q.set('geo', geo ? '1' : '0');
const qs = q.toString();
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
replaceState: true,
noScroll: true,
keepFocus: true
});
}
let queryKey = $derived(page.url.searchParams.toString());
$effect(() => {
queryKey;
if (typeof window === 'undefined') return;
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
refreshSeconds = parseRefresh(page.url.searchParams.get('refresh'));
geo = parse01(page.url.searchParams.get('geo'), true);
aliasesInput = aliases;
refreshInput = String(refreshSeconds);
restartPolling();
void load();
});
let staleSeconds = $derived.by(() =>
lastSuccessAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastSuccessAtMs) / 1000))
);
let isStale = $derived.by(() =>
staleSeconds == null ? true : staleSeconds > (refreshSeconds > 0 ? refreshSeconds * 2 : 60)
);
onMount(() => {
staleTimer = setInterval(() => {
nowMs = Date.now();
}, 1000);
return () => {
if (pollTimer) clearInterval(pollTimer);
if (staleTimer) clearInterval(staleTimer);
};
});
</script>
<div class="mb-6 flex flex-wrap items-end justify-between gap-4">
@@ -301,17 +378,50 @@
<p class="text-sm text-muted-foreground">Снимок active/recent с шлюза; GeoIP из конфига шлюза.</p>
</div>
<div class="flex items-center gap-2">
<Badge variant={isStale ? 'secondary' : 'default'}>
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
</Badge>
<label class="flex cursor-pointer items-center gap-2 text-sm">
<input type="checkbox" bind:checked={geo} onchange={load} class="rounded border" />
<input
type="checkbox"
bind:checked={geo}
onchange={() => void applyControls()}
class="rounded border"
/>
GeoIP
</label>
<Button variant="outline" size="sm" onclick={load} disabled={loading}>
<Button variant="outline" size="sm" onclick={() => void load()} disabled={loading}>
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
Обновить
</Button>
</div>
</div>
<div class="mb-4 flex flex-wrap items-end gap-2">
<label class="text-xs text-muted-foreground">
Aliases
<input
type="text"
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
bind:value={aliasesInput}
placeholder="node-a,node-b"
/>
</label>
<label class="text-xs text-muted-foreground">
Refresh (sec)
<input
type="number"
min="0"
max="300"
class="mt-1 h-9 w-28 rounded-md border bg-background px-2 text-sm"
bind:value={refreshInput}
/>
</label>
<Button variant="outline" size="sm" onclick={() => void applyControls()} disabled={loading}>
Apply
</Button>
</div>
{#if partial}
<Alert class="mb-4">
<AlertTitle>Частичные данные</AlertTitle>
+348
View File
@@ -0,0 +1,348 @@
<script lang="ts">
import { onMount } from 'svelte';
import { page } from '$app/state';
import { goto } from '$app/navigation';
import {
liveEventsUrl,
type IncidentItem,
type IncidentSeverity,
type IncidentStatus
} from '$lib/api/client.js';
import * as Card from '$lib/components/ui/card/index.js';
import { Alert, AlertDescription, AlertTitle } from '$lib/components/ui/alert/index.js';
import { Badge } from '$lib/components/ui/badge/index.js';
import { Button } from '$lib/components/ui/button/index.js';
import * as Table from '$lib/components/ui/table/index.js';
import RefreshCwIcon from '@lucide/svelte/icons/refresh-cw';
type LiveStatus = 'healthy' | 'degraded' | 'critical';
type LiveSnapshot = {
type?: string;
timestamp?: string;
status?: LiveStatus;
partial?: boolean;
incidents?: IncidentItem[];
counts?: {
total?: number;
critical?: number;
warning?: number;
info?: number;
};
aliases_used?: string[];
};
let aliases = $state('');
let aliasesInput = $state('');
let err = $state<string | null>(null);
let connected = $state(false);
let reconnectInSec = $state<number | null>(null);
let reconnectAttempt = $state(0);
let snapshot = $state<LiveSnapshot | null>(null);
let lastEventAtMs = $state<number | null>(null);
let nowMs = $state(Date.now());
let reconnectTimer: ReturnType<typeof setTimeout> | null = null;
let staleTimer: ReturnType<typeof setInterval> | null = null;
let es: EventSource | null = null;
function severityVariant(sev: IncidentSeverity): 'default' | 'secondary' | 'destructive' | 'outline' {
if (sev === 'critical') return 'destructive';
if (sev === 'warning') return 'secondary';
return 'outline';
}
function statusVariant(st: LiveStatus | undefined): 'default' | 'secondary' | 'destructive' {
if (st === 'critical') return 'destructive';
if (st === 'degraded') return 'secondary';
return 'default';
}
function normIncident(v: unknown): IncidentItem | null {
if (!v || typeof v !== 'object') return null;
const x = v as Record<string, unknown>;
const id = typeof x.id === 'string' ? x.id : '';
if (!id) return null;
const severity = (x.severity ?? 'info') as IncidentSeverity;
const status = (x.status ?? 'firing') as IncidentStatus;
return {
id,
kind: typeof x.kind === 'string' ? x.kind : '',
severity,
status,
title: typeof x.title === 'string' ? x.title : id,
summary: typeof x.summary === 'string' ? x.summary : '',
affected_aliases: Array.isArray(x.affected_aliases)
? x.affected_aliases.filter((a): a is string => typeof a === 'string')
: [],
metric_name: typeof x.metric_name === 'string' ? x.metric_name : undefined,
metric_value: typeof x.metric_value === 'number' ? x.metric_value : undefined,
metric_threshold: typeof x.metric_threshold === 'number' ? x.metric_threshold : undefined,
actions: Array.isArray(x.actions)
? x.actions
.filter((a) => a && typeof a === 'object')
.map((a) => a as Record<string, unknown>)
.filter((a) => typeof a.label === 'string' && typeof a.href === 'string')
.map((a) => ({ label: String(a.label), href: String(a.href) }))
: []
};
}
function parseSnapshot(raw: unknown): LiveSnapshot | null {
if (!raw || typeof raw !== 'object') return null;
const x = raw as Record<string, unknown>;
const incidents = Array.isArray(x.incidents) ? x.incidents.map(normIncident).filter(Boolean) : [];
const countsRaw = x.counts && typeof x.counts === 'object' ? (x.counts as Record<string, unknown>) : {};
return {
type: typeof x.type === 'string' ? x.type : undefined,
timestamp: typeof x.timestamp === 'string' ? x.timestamp : undefined,
status:
x.status === 'healthy' || x.status === 'degraded' || x.status === 'critical'
? x.status
: 'healthy',
partial: !!x.partial,
incidents: incidents as IncidentItem[],
counts: {
total: typeof countsRaw.total === 'number' ? countsRaw.total : incidents.length,
critical: typeof countsRaw.critical === 'number' ? countsRaw.critical : undefined,
warning: typeof countsRaw.warning === 'number' ? countsRaw.warning : undefined,
info: typeof countsRaw.info === 'number' ? countsRaw.info : undefined
},
aliases_used: Array.isArray(x.aliases_used)
? x.aliases_used.filter((a): a is string => typeof a === 'string')
: []
};
}
function clearReconnectTimer() {
if (!reconnectTimer) return;
clearTimeout(reconnectTimer);
reconnectTimer = null;
}
function closeStream() {
if (es) {
es.close();
es = null;
}
connected = false;
}
function scheduleReconnect() {
clearReconnectTimer();
const delayMs = Math.min(10_000, 1000 * 2 ** Math.min(8, reconnectAttempt));
reconnectAttempt += 1;
reconnectInSec = Math.ceil(delayMs / 1000);
const target = Date.now() + delayMs;
reconnectTimer = setTimeout(() => {
reconnectInSec = null;
connectStream();
}, delayMs);
const countdown = setInterval(() => {
const left = Math.max(0, Math.ceil((target - Date.now()) / 1000));
reconnectInSec = left;
if (left <= 0) clearInterval(countdown);
}, 250);
}
function connectStream() {
closeStream();
clearReconnectTimer();
err = null;
const url = liveEventsUrl({ aliases: aliases || undefined });
const next = new EventSource(url);
next.onopen = () => {
connected = true;
reconnectAttempt = 0;
reconnectInSec = null;
};
next.onmessage = (ev) => {
try {
const parsed = parseSnapshot(JSON.parse(ev.data));
if (!parsed) return;
snapshot = parsed;
lastEventAtMs = Date.now();
err = null;
connected = true;
} catch (e) {
err = `Невалидный snapshot: ${String(e)}`;
}
};
next.addEventListener('snapshot', (ev) => {
try {
const msg = ev as MessageEvent<string>;
const parsed = parseSnapshot(JSON.parse(msg.data));
if (!parsed) return;
snapshot = parsed;
lastEventAtMs = Date.now();
err = null;
connected = true;
} catch (e) {
err = `Невалидный snapshot: ${String(e)}`;
}
});
next.onerror = () => {
connected = false;
err = 'Поток SSE разорван, переподключение...';
next.close();
if (es === next) es = null;
scheduleReconnect();
};
es = next;
}
async function applyControls() {
const q = new URLSearchParams(page.url.searchParams);
const normalizedAliases = aliasesInput.trim();
if (normalizedAliases) q.set('aliases', normalizedAliases);
else q.delete('aliases');
const qs = q.toString();
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
replaceState: true,
noScroll: true,
keepFocus: true
});
}
let queryKey = $derived(page.url.searchParams.toString());
$effect(() => {
queryKey;
if (typeof window === 'undefined') return;
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
aliasesInput = aliases;
connectStream();
});
let staleSeconds = $derived.by(() =>
lastEventAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastEventAtMs) / 1000))
);
let isStale = $derived.by(() => staleSeconds == null || staleSeconds > 12);
onMount(() => {
staleTimer = setInterval(() => {
nowMs = Date.now();
}, 1000);
return () => {
closeStream();
clearReconnectTimer();
if (staleTimer) clearInterval(staleTimer);
};
});
</script>
<div class="mb-6 flex flex-wrap items-center justify-between gap-4">
<div>
<h1 class="text-2xl font-semibold tracking-tight">Live</h1>
<p class="text-sm text-muted-foreground">SSE поток инцидентов и статуса флота.</p>
</div>
<div class="flex items-center gap-2">
<Badge variant={connected ? 'default' : 'secondary'}>
{connected ? 'connected' : 'disconnected'}
</Badge>
<Badge variant={isStale ? 'secondary' : 'default'}>
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
</Badge>
<Button variant="outline" size="sm" onclick={() => connectStream()}>
<RefreshCwIcon class="mr-1 size-4" />
Reconnect
</Button>
</div>
</div>
<div class="mb-4 flex flex-wrap items-end gap-2">
<label class="text-xs text-muted-foreground">
Aliases
<input
type="text"
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
bind:value={aliasesInput}
placeholder="node-a,node-b"
/>
</label>
<Button variant="outline" size="sm" onclick={() => void applyControls()}>
Apply
</Button>
</div>
{#if err}
<Alert class="mb-4">
<AlertTitle>Live stream</AlertTitle>
<AlertDescription>
{err}
{#if reconnectInSec != null}
Переподключение через {reconnectInSec}s.
{/if}
</AlertDescription>
</Alert>
{/if}
<div class="mb-4 grid gap-4 sm:grid-cols-3">
<Card.Root>
<Card.Header class="pb-2">
<Card.Description>Общий статус</Card.Description>
<Card.Title>
<Badge variant={statusVariant(snapshot?.status)}>
{snapshot?.status ?? 'healthy'}
</Badge>
</Card.Title>
</Card.Header>
</Card.Root>
<Card.Root>
<Card.Header class="pb-2">
<Card.Description>Partial</Card.Description>
<Card.Title class="text-2xl tabular-nums">{snapshot?.partial ? 'yes' : 'no'}</Card.Title>
</Card.Header>
</Card.Root>
<Card.Root>
<Card.Header class="pb-2">
<Card.Description>Timestamp</Card.Description>
<Card.Title class="text-sm">
{snapshot?.timestamp ? new Date(snapshot.timestamp).toLocaleString() : '—'}
</Card.Title>
</Card.Header>
</Card.Root>
</div>
<Card.Root>
<Card.Header>
<Card.Title>Последние incidents из snapshot</Card.Title>
<Card.Description>Всего: {snapshot?.counts?.total ?? snapshot?.incidents?.length ?? 0}</Card.Description>
</Card.Header>
<Card.Content class="p-0">
<Table.Root>
<Table.Header>
<Table.Row>
<Table.Head>Severity</Table.Head>
<Table.Head>Title</Table.Head>
<Table.Head>Summary</Table.Head>
<Table.Head>Aliases</Table.Head>
</Table.Row>
</Table.Header>
<Table.Body>
{#if (snapshot?.incidents?.length ?? 0) === 0}
<Table.Row>
<Table.Cell colspan={4} class="p-4 text-center text-muted-foreground">
Инцидентов нет
</Table.Cell>
</Table.Row>
{:else}
{#each snapshot?.incidents ?? [] as item (item.id)}
<Table.Row>
<Table.Cell>
<Badge variant={severityVariant(item.severity)}>{item.severity}</Badge>
</Table.Cell>
<Table.Cell class="font-medium">{item.title}</Table.Cell>
<Table.Cell class="max-w-[380px] text-sm text-muted-foreground">{item.summary}</Table.Cell>
<Table.Cell>
<div class="flex flex-wrap gap-1">
{#each item.affected_aliases ?? [] as alias (alias)}
<Badge variant="outline">{alias}</Badge>
{/each}
</div>
</Table.Cell>
</Table.Row>
{/each}
{/if}
</Table.Body>
</Table.Root>
</Card.Content>
</Card.Root>
+121 -4
View File
@@ -1,9 +1,11 @@
<script lang="ts">
import { onMount } from 'svelte';
import { page } from '$app/state';
import { goto } from '$app/navigation';
import { fetchAggUsers, ApiError } from '$lib/api/client.js';
import type { components } from '$lib/api/aggregate.gen.js';
import { formatMiB } from '$lib/format.js';
import { Badge } from '$lib/components/ui/badge/index.js';
import { Button } from '$lib/components/ui/button/index.js';
import * as Table from '$lib/components/ui/table/index.js';
import * as Card from '$lib/components/ui/card/index.js';
@@ -16,14 +18,39 @@
let rows = $state<components['schemas']['UsersRow'][]>([]);
let partial = $state(false);
let includeLinks = $state(false);
let aliases = $state('');
let aliasesInput = $state('');
let refreshSeconds = $state(30);
let refreshInput = $state('30');
let lastSuccessAtMs = $state<number | null>(null);
let nowMs = $state(Date.now());
let pollTimer: ReturnType<typeof setInterval> | null = null;
let staleTimer: ReturnType<typeof setInterval> | null = null;
function parseRefresh(raw: string | null): number {
if (raw == null || raw.trim() === '') return 30;
const n = Number(raw);
if (!Number.isFinite(n)) return 30;
if (n === 0) return 0;
return Math.max(10, Math.min(300, Math.floor(n)));
}
function parse01(raw: string | null, fallback: boolean): boolean {
if (raw == null || raw === '') return fallback;
return raw === '1';
}
async function load() {
loading = true;
err = null;
try {
const usersEnv = await fetchAggUsers({ include_links: includeLinks });
const usersEnv = await fetchAggUsers({
aliases: aliases || undefined,
include_links: includeLinks
});
partial = !!usersEnv.partial;
rows = usersEnv.data ?? [];
lastSuccessAtMs = Date.now();
} catch (e) {
err = e instanceof ApiError ? e.message : String(e);
} finally {
@@ -31,7 +58,64 @@
}
}
onMount(load);
function restartPolling() {
if (pollTimer) {
clearInterval(pollTimer);
pollTimer = null;
}
if (refreshSeconds > 0) {
pollTimer = setInterval(() => {
void load();
}, refreshSeconds * 1000);
}
}
async function applyControls() {
const q = new URLSearchParams(page.url.searchParams);
const normalizedAliases = aliasesInput.trim();
const parsedRefresh = parseRefresh(refreshInput);
if (normalizedAliases) q.set('aliases', normalizedAliases);
else q.delete('aliases');
if (parsedRefresh === 30) q.delete('refresh');
else q.set('refresh', String(parsedRefresh));
q.set('include_links', includeLinks ? '1' : '0');
const qs = q.toString();
await goto(`${page.url.pathname}${qs ? `?${qs}` : ''}`, {
replaceState: true,
noScroll: true,
keepFocus: true
});
}
let queryKey = $derived(page.url.searchParams.toString());
$effect(() => {
queryKey;
if (typeof window === 'undefined') return;
aliases = page.url.searchParams.get('aliases')?.trim() ?? '';
refreshSeconds = parseRefresh(page.url.searchParams.get('refresh'));
includeLinks = parse01(page.url.searchParams.get('include_links'), false);
aliasesInput = aliases;
refreshInput = String(refreshSeconds);
restartPolling();
void load();
});
let staleSeconds = $derived.by(() =>
lastSuccessAtMs == null ? null : Math.max(0, Math.floor((nowMs - lastSuccessAtMs) / 1000))
);
let isStale = $derived.by(() =>
staleSeconds == null ? true : staleSeconds > (refreshSeconds > 0 ? refreshSeconds * 2 : 60)
);
onMount(() => {
staleTimer = setInterval(() => {
nowMs = Date.now();
}, 1000);
return () => {
if (pollTimer) clearInterval(pollTimer);
if (staleTimer) clearInterval(staleTimer);
};
});
function copy(text: string) {
const value = String(text ?? '');
@@ -76,17 +160,50 @@
</p>
</div>
<div class="flex items-center gap-2">
<Badge variant={isStale ? 'secondary' : 'default'}>
{isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if}
</Badge>
<label class="flex cursor-pointer items-center gap-2 text-sm">
<input type="checkbox" bind:checked={includeLinks} onchange={load} class="rounded border" />
<input
type="checkbox"
bind:checked={includeLinks}
onchange={() => void applyControls()}
class="rounded border"
/>
Ссылки tg://proxy
</label>
<Button variant="outline" size="sm" onclick={load} disabled={loading}>
<Button variant="outline" size="sm" onclick={() => void load()} disabled={loading}>
<RefreshCwIcon class="mr-1 size-4 {loading ? 'animate-spin' : ''}" />
Обновить
</Button>
</div>
</div>
<div class="mb-4 flex flex-wrap items-end gap-2">
<label class="text-xs text-muted-foreground">
Aliases
<input
type="text"
class="mt-1 h-9 w-56 rounded-md border bg-background px-2 text-sm"
bind:value={aliasesInput}
placeholder="node-a,node-b"
/>
</label>
<label class="text-xs text-muted-foreground">
Refresh (sec)
<input
type="number"
min="0"
max="300"
class="mt-1 h-9 w-28 rounded-md border bg-background px-2 text-sm"
bind:value={refreshInput}
/>
</label>
<Button variant="outline" size="sm" onclick={() => void applyControls()} disabled={loading}>
Apply
</Button>
</div>
{#if partial}
<Alert class="mb-4">
<AlertTitle>Частичные данные</AlertTitle>