observability: alert-relay (HyperDX webhook -> Telegram) + Telegram-webhook ja 'Teenus maas' alert; docs

dev/LOEMIND_observability.md

@@ -113,17 +113,30 @@ Access key: `docker exec clickstack mongo --quiet hyperdx --eval 'print(db.users
   "groupBy":"name"}}
 ```
 
-## Alertid
-Alert vajab **teavituskanalit** (`channel`/`channels`) — HyperDX webhook. Ilma kanalita
-API tagastab „Invalid input". Seadistus: UI → Alerts → New Alert, või Webhooks all
-veebikonks (nt n8n → Telegram-relay).
-API: `POST /api/v2/alerts`, kohustuslikud `interval`, `threshold`, `thresholdType`,
-`source` (`inline`|`tile`|`saved_search`) + `channel`.
-
-Näidikud, mida tasub alertida:
-- `ohmu_service_up < 1` (teenus maas) — `thresholdType:"below"`, `threshold:1`, `interval:"5m"`
-- `node_memory_MemAvailable_bytes` liiga madal (RAM otsas)
-- `ohmu_container_restarts` kasvab (konteineri restart-loop)
+## Alertid (Telegram)
+Rakendatud: **HyperDX webhook → `alert-relay` → Telegram**.
+- **`alert-relay`** konteiner (`dev/config/alert-relay/alert_relay.py`, `python:3.12-slim`, port 9099):
+  võtab vastu HyperDX „generic" webhook POST-id ja saadab Telegrami. Token `dev/.env`-st
+  (`TELEGRAM_BOT_TOKEN`), chat `5036685258`.
+- **Webhook:** `Telegram (ohmu)` → `http://alert-relay:9099/` (id `6ab760240c9282a5b0fde124`).
+- **Alert:** „Teenus maas (ohmu_service_up)" → dashboardi paneel „Teenused üleval",
+  `thresholdType=below`, `threshold=1`, `interval=5m`, kanal webhook (id `6ab7603c0c9282a5b0fde12d`).
+
+**Oluline API-piirang:** väline API v2 (`POST /api/v2/alerts`) toetab ainult
+`source: "tile"` või `"saved_search"` — **`inline` alert on ainult sisemise API jaoks**.
+Paneeli-alert viitab: `{source:"tile", tileId, dashboardId}`; kanal:
+`{"type":"webhook","webhookId":"..."}`.
+
+**Testimine:**
+```bash
+# relay otse (hostilt pole 9099 avaldatud; kasuta konteinerit võrgus):
+docker run --rm --network main_ai_network curlimages/curl:latest -s \
+  -X POST http://alert-relay:9099/ -d '{"name":"test","state":"ALERT"}'
+```
+
+Tüüpilised täiendavad alertid (sama muster: tile + webhook + below/above):
+- `node_memory_MemAvailable_bytes` < X (RAM otsas)
+- `ohmu_container_restarts` > 0 (restart-loop)
 - `llamacpp:kv_cache_usage_ratio` > 0.9
 
 ## Seotud

dev/config/alert-relay/alert_relay.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""HyperDX alert-webhook → Telegram relay (ohmu).
+
+Võtab vastu HyperDX "generic" webhook POST-i ja saadab sisu Telegrami.
+Kasutab ainult standardteeki (ei vaja pip-pakette).
+
+Env: TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID (vaikimisi 5036685258)
+Kuulab: 0.0.0.0:9099
+"""
+from __future__ import annotations
+
+import json
+import os
+import urllib.parse
+import urllib.request
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN", "").strip()
+CHAT = os.environ.get("TELEGRAM_CHAT_ID", "5036685258")
+PORT = int(os.environ.get("PORT", "9099"))
+
+
+def send(text: str) -> None:
+    if not TOKEN:
+        print("[relay] TELEGRAM_BOT_TOKEN puudub", flush=True)
+        return
+    data = urllib.parse.urlencode({"chat_id": CHAT, "text": text[:4000], "parse_mode": "Markdown"}).encode()
+    req = urllib.request.Request(f"https://api.telegram.org/bot{TOKEN}/sendMessage", data=data)
+    try:
+        with urllib.request.urlopen(req, timeout=20) as r:
+            r.read()
+    except Exception as e:  # noqa: BLE001
+        print("[relay] Telegram viga:", str(e)[:200], flush=True)
+
+
+def format_msg(data: dict) -> str:
+    def pick(*keys):
+        for k in keys:
+            v = data.get(k) or (data.get("alert") or {}).get(k)
+            if v not in (None, ""):
+                return v
+        return None
+
+    name = pick("name", "alertName", "title") or "HyperDX alert"
+    state = pick("state", "status", "event") or "?"
+    icon = {"ALERT": "🔴", "alert": "🔴", "RESOLVED": "✅", "resolved": "✅", "OK": "✅"}.get(str(state), "⚠️")
+    lines = [f"{icon} *{name}*", f"Seis: `{state}`"]
+    msg = pick("message", "note", "description")
+    if msg:
+        lines.append(str(msg)[:800])
+    if not msg:
+        lines.append("```\n" + json.dumps(data, ensure_ascii=False)[:1200] + "\n```")
+    return "\n".join(lines)
+
+
+class H(BaseHTTPRequestHandler):
+    def do_POST(self):  # noqa: N802
+        n = int(self.headers.get("Content-Length", "0") or 0)
+        raw = self.rfile.read(n) if n else b"{}"
+        try:
+            data = json.loads(raw or b"{}")
+        except Exception:  # noqa: BLE001
+            data = {"message": raw.decode("utf-8", "replace")[:1000]}
+        send(format_msg(data if isinstance(data, dict) else {"message": str(data)}))
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()
+        self.wfile.write(b'{"ok":true}')
+
+    def do_GET(self):  # noqa: N802
+        self.send_response(200)
+        self.end_headers()
+        self.wfile.write(b"alert-relay ok")
+
+    def log_message(self, *args):  # vaikimisi logi maha
+        return
+
+
+if __name__ == "__main__":
+    print(f"[relay] kuulan :{PORT} -> Telegram {CHAT}", flush=True)
+    HTTPServer(("0.0.0.0", PORT), H).serve_forever()

dev/docker-compose.yml

@@ -199,6 +199,22 @@ services:
     networks:
       - main_ai_network
 
+  # HyperDX alert-webhook → Telegram relay (generic webhook vastuvõtja)
+  alert-relay:
+    image: python:3.12-slim
+    container_name: alert-relay
+    restart: unless-stopped
+    mem_limit: 64m
+    command: ["python", "/app/alert_relay.py"]
+    environment:
+      - TZ=Europe/Tallinn
+      - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
+      - TELEGRAM_CHAT_ID=5036685258
+    volumes:
+      - ./config/alert-relay/alert_relay.py:/app/alert_relay.py:ro
+    networks:
+      - main_ai_network
+
 networks:
   main_ai_network:
     external: true