Enhance API and UI for incident management and live updates

- Added a new endpoint `/api/agg/incidents` to provide a normalized snapshot of incidents for fleet triage, including severity and recommended actions. - Implemented live event streaming via `/api/live/events` for real-time updates on fleet status and incidents, enhancing observability. - Updated the Web UI to include dedicated sections for incidents and live updates, improving user navigation and access to critical information. - Enhanced API documentation to reflect new endpoints and their functionalities, ensuring clarity for developers and users.
2026-03-30 19:17:29 +07:00
parent af11a49c81
commit 8c8ccce6ee
18 changed files with 1655 additions and 28 deletions
@@ -33,6 +33,7 @@
 | GET | `/api/agg/users` | Объединённый список пользователей с `by_server`, суммарным `total_megabytes` и **смерженными лимитами** (см. ниже). |
 | GET | `/api/agg/user/{username}` | Один пользователь в том же формате, что элементы `/api/agg/users` (без списка всех). Имя в пути: `[A-Za-z0-9_.-]+`. Ответ **`404`**, если пользователь не найден ни на одном успешном upstream. |
 | GET | `/api/agg/fleet-status` | По каждому алиасу: параллельно health + system/info; в `data.servers[]` — статусы подзапросов и тела `health` / `system_info` при успехе. См. [AGGREGATE_OPENAPI.yaml](AGGREGATE_OPENAPI.yaml). |
+| GET | `/api/agg/incidents` | Нормализованный snapshot инцидентов для triage-панели: `critical/warning/info`, `affected_aliases`, рекомендуемые `actions` (runbook/deep links), счётчики по severity. |

 Все методы — **GET**; действует тот же whitelist, что и для остального API шлюза.

@@ -57,6 +58,18 @@
 | `include_links` | `users`, `user/…` | `true` — добавить сгенерированные `tg://proxy` ссылки (берётся первая успешная запись по пользователю). |
 | `min_total_megabytes` | `users` | Порог суммарного трафика пользователя в MiB (строго больше 0). |
 | `min_total_octets` | `users` | Устаревший вариант порога в октетах (если задан `min_total_megabytes`, он приоритетнее). |
+| `aliases` | `incidents` | Список алиасов через запятую; позволяет строить incidents snapshot по выбранной группе нод. |
+
+## Live stream (SSE)
+
+Для оперативного режима NOC доступен поток событий:
+
+- **`GET /api/live/events`** (`text/event-stream`)
+- query: `aliases` (опционально, как в `/api/agg/*`)
+- событие: `event: snapshot`
+- payload: JSON со статусом флота (`healthy/degraded/critical`), `partial`, и массивом `incidents`
+
+Поток рассчитан на UI-клиент с авто-reconnect (на фронте используется экспоненциальный backoff).

 ## Конфигурация (опционально)

@@ -99,4 +112,8 @@ curl -sS "http://127.0.0.1:8080/api/agg/unique-ips"
 curl -sS "http://127.0.0.1:8080/api/agg/users?include_links=false&min_total_megabytes=1"
 curl -sS "http://127.0.0.1:8080/api/agg/fleet-status"
 curl -sS "http://127.0.0.1:8080/api/agg/user/myuser?aliases=gt1"
+curl -sS "http://127.0.0.1:8080/api/agg/incidents?aliases=gt1,gt2"
+
+# SSE поток snapshot-событий (пример с curl)
+curl -N "http://127.0.0.1:8080/api/live/events?aliases=gt1,gt2"
 ```
@@ -106,6 +106,18 @@ paths:
            application/json:
              schema: { $ref: '#/components/schemas/AggEnvelopeFleetStatus' }

+  /api/agg/incidents:
+    get:
+      summary: Нормализованный snapshot инцидентов по флоту
+      parameters:
+        - $ref: '#/components/parameters/aliases'
+      responses:
+        '200':
+          description: OK
+          content:
+            application/json:
+              schema: { $ref: '#/components/schemas/AggEnvelopeIncidents' }
+
 components:
  parameters:
    aliases:
@@ -173,6 +185,13 @@ components:
          properties:
            data: { $ref: '#/components/schemas/FleetStatusData' }

+    AggEnvelopeIncidents:
+      allOf:
+        - $ref: '#/components/schemas/AggSuccessBase'
+        - type: object
+          properties:
+            data: { $ref: '#/components/schemas/IncidentsData' }
+
    TrafficRow:
      type: object
      properties:
@@ -297,3 +316,43 @@ components:
        servers_total: { type: integer }
        servers_all_ok: { type: integer }
        servers_failed: { type: integer }
+
+    IncidentAction:
+      type: object
+      properties:
+        label: { type: string }
+        href: { type: string }
+
+    IncidentItem:
+      type: object
+      properties:
+        id: { type: string }
+        kind: { type: string }
+        severity:
+          type: string
+          enum: [info, warning, critical]
+        status:
+          type: string
+          enum: [firing]
+        title: { type: string }
+        summary: { type: string }
+        affected_aliases:
+          type: array
+          items: { type: string }
+        metric_name: { type: string }
+        metric_value: { type: number, format: float }
+        metric_threshold: { type: number, format: float }
+        actions:
+          type: array
+          items: { $ref: '#/components/schemas/IncidentAction' }
+
+    IncidentsData:
+      type: object
+      properties:
+        items:
+          type: array
+          items: { $ref: '#/components/schemas/IncidentItem' }
+        total: { type: integer }
+        critical_total: { type: integer }
+        warning_total: { type: integer }
+        info_total: { type: integer }
@@ -0,0 +1,73 @@
+# Incident Rollout Playbook
+
+## Scope
+
+This playbook describes staged rollout for:
+
+- `/api/agg/incidents`
+- `/api/live/events` (SSE snapshots)
+- UI pages `/incidents` and `/live`
+- Live polling controls and stale indicators on key pages
+
+## Stage 0 - Baseline Capture (2-3 days)
+
+- Record current MTTD and MTTR from on-call logs.
+- Record manual refresh usage on main pages.
+- Save top recurring failure patterns (degraded nodes, read-only modes, bad connections spikes).
+
+Outputs:
+
+- baseline MTTD / MTTR
+- top 5 incident categories by frequency
+
+## Stage 1 - Shadow Mode (3-5 days)
+
+- Enable incidents and live pages for operators.
+- Do not change paging/escalation yet.
+- Compare incident feed against existing monitoring and mark false positives.
+
+Targets:
+
+- false positive ratio < 20%
+- no increase in upstream load beyond acceptable budget
+
+## Stage 2 - Assisted Triage (1 week)
+
+- Use `/incidents` as primary triage board.
+- Require owner + ack for active critical incidents.
+- Use runbook links from incident items.
+
+Targets:
+
+- ack coverage for critical incidents >= 90%
+- owner coverage for critical incidents >= 90%
+
+## Stage 3 - Policy Tuning (ongoing)
+
+- Adjust thresholds:
+  - `bad_connections_warn` (default 1000)
+  - `bad_connections_high` (default 10000)
+- Review alert fatigue weekly.
+- Promote stable thresholds into documented policy.
+
+## KPI Tracking
+
+- MTTD (minutes): incident first observed -> first ack
+- MTTR (minutes): incident first observed -> resolved
+- Stale time share: percentage of time live views are stale
+- Manual refresh share: manual refresh / total data update actions
+
+## Fast Rollback
+
+If noise or load is excessive:
+
+1. disable auto-refresh by setting `refresh=0` in shared ops links
+2. switch operators back to dashboard summary only
+3. keep `/api/agg/incidents` for diagnostics while disabling SSE consumers
+
+## Weekly Review Template
+
+- KPI delta (MTTD, MTTR) vs baseline
+- top noisy rules
+- incidents with missing owner/ack
+- policy changes applied this week
@@ -0,0 +1,49 @@
+# Operations Baseline (Telemt Panel)
+
+## Purpose
+
+This document defines the baseline operating model and target SLO metrics for the Telemt response panel.
+It is used to measure impact of live updates, incidents workflow, and observability improvements.
+
+## Current Baseline (Before Rollout)
+
+- Dashboard refresh mode: mostly manual (`Refresh` buttons on key pages).
+- Unified incident queue: not present.
+- Alert ownership and acknowledgement flow: not present.
+- MTTR tracking: not formalized in product UI.
+- MTTD tracking: not formalized in product UI.
+- Cross-page filtering consistency: partial, per-page controls only.
+
+## Baseline Risks
+
+- Slow detection when operators do not refresh at the right time.
+- High context switching cost between pages during incidents.
+- No single place to triage partial/degraded node states.
+- Weak evidence trail for post-incident analysis.
+
+## SLO Targets (Phase Goals)
+
+- MTTD reduction: at least 30% versus baseline.
+- MTTR reduction: at least 25% versus baseline.
+- Manual refresh share during incident windows: under 10%.
+- Incident cards with `owner + ack + runbook`: over 90%.
+
+## Measurement Inputs
+
+- Gateway API:
+  - `/api/agg/fleet-status`
+  - `/api/agg/summary`
+  - `/api/agg/incidents` (planned/implemented in this rollout)
+- Gateway metrics endpoint:
+  - `/metrics`
+- Frontend telemetry (local panel interaction metrics):
+  - refresh actions
+  - ack/resolve actions
+  - stale/live state durations
+
+## Rollout Validation Checklist
+
+- [ ] Baseline values captured before enabling auto-refresh.
+- [ ] Incident flow tested with simulated degraded upstream.
+- [ ] Alert noise review completed after first week.
+- [ ] MTTD/MTTR comparison published for phase review.