|
|
@@ -1,40 +1,54 @@
|
|
|
# Observability (ClickStack / HyperDX) — ohmu
|
|
|
|
|
|
-**Loodud 2026-09-25.** ClickStack = HyperDX UI + (väline) ClickHouse + kaasas olev
|
|
|
-OTel-kollektor. Kogu grupp on **always-on** (ei ole enam `experimental`).
|
|
|
+**Loodud 2026-09-25; migreeritud C-full arhitektuurile 2026-10-03.** ClickStack = HyperDX UI (app-only)
|
|
|
++ eraldi MongoDB (rakenduse olek) + ametlik ClickStack OTel-kollektor + (väline) ClickHouse.
|
|
|
+Kogu grupp on **always-on**.
|
|
|
|
|
|
## Ligipääs
|
|
|
-- UI: **http://100.87.1.24:8124/clickstack/search** (läbi `clickstack-proxy`)
|
|
|
-- UI otse: http://100.87.1.24:18080
|
|
|
+- UI (kanooniline): **http://100.87.1.24:18080**
|
|
|
+- `8124/clickstack` → **302 → 18080** (ühtlustatud; ClickHouse'i bundle'itud local-mode UI-d ei kasutata)
|
|
|
- Kasutaja: `admin@odamus.com` / `Dev_clickhouse_2026!`
|
|
|
- ClickHouse (väline, `clickhouse` konteiner): HTTP 8125, native 9004 (Tailscale-only)
|
|
|
|
|
|
-## Arhitektuur (andmevoog)
|
|
|
+## Arhitektuur (andmevoog) — C-full (2026-10-03)
|
|
|
```
|
|
|
node-exporter ─┐
|
|
|
-cadvisor ──────┼─► otel-collector (scrape 60s) ─► clickstack:4317 (kaasas kollektor) ─► ClickHouse (default.otel_*)
|
|
|
-llama.cpp ─────┘ (main_ai_network) (opampsupervisor)
|
|
|
+cadvisor ──────┼─► otel-collector (clickstack-otel-collector:2) ─┐
|
|
|
+llama.cpp ─────┘ prometheus (custom.config) │
|
|
|
+konteinerilogid ─► filelog (custom.config) ──────────────────────┼─► ClickHouse (default.otel_*)
|
|
|
+app/SDK OTLP ────► OTLP receiver (base, OpAMP kaudu app'ist) ────┘
|
|
|
+ ▲ base config: app OpAMP (:4320)
|
|
|
+app (HyperDX, app-only) ── MONGO_URI ──► clickstack-db (mongo:5.0.32)
|
|
|
```
|
|
|
-- `otel-collector` (contrib) skrape'ib node-exporter, cadvisor ja **llama.cpp** (`host.docker.internal:8070/metrics`).
|
|
|
-- Kaasas olev kollektor (clickstack konteineris) võtab configu API-lt (opamp, port 4320) ja kirjutab ClickHouse'i `otel_*` tabelitesse.
|
|
|
-- **Kriitiline:** `clickstack-entry.sh` peab käivitama `/otel-entrypoint.sh /usr/local/bin/opampsupervisor` — muidu OTLP andmed ei püsi.
|
|
|
+- `clickstack` = **app-only** (`docker.hyperdx.io/hyperdx/hyperdx:2.37.0`) — UI+API, OpAMP server (4320), `MONGO_URI`.
|
|
|
+- `clickstack-db` = **mongo:5.0.32** — rakenduse olek (sources/dashboards/alerts/users), migreeritud all-in-one'ist (mongodump/restore).
|
|
|
+- `otel-collector` = **ametlik** `docker.clickhouse.com/clickhouse/clickstack-otel-collector:2` — base config tuleb app'ist (OpAMP), skeem/rollupid; `CUSTOM_OTELCOL_CONFIG_FILE` lisab hosti-meetrikad + konteinerilogid.
|
|
|
+- ClickHouse on **väline** (eraldi konteiner).
|
|
|
+- ⚠️ UI-chunkide patchid (heatmap/CTE/valideerimine) on seotud app-versiooniga **2.37.0** — hoia app pin'itud.
|
|
|
+- ℹ️ Varem oli `hyperdx-all-in-one` (dubleeris ClickHouse+Mongo). Migratsiooni backup:
|
|
|
+ `/data/backups/clickstack-migration-<TS>/` (compose/config + mongodump).
|
|
|
|
|
|
## ClickHouse tabelid (default)
|
|
|
-`otel_logs`, `otel_traces`, `otel_metrics_gauge|sum|histogram|exponential_histogram|summary` + rollupid.
|
|
|
-Skeem luuakse automaatselt schema-seed'iga (goose).
|
|
|
+`otel_logs`, `otel_traces`, `otel_metrics_*` + rollupid, `hyperdx_sessions` (session replays).
|
|
|
+Skeem luuakse ametliku kollektori schema-seed'iga.
|
|
|
|
|
|
## Konfiguratsioon
|
|
|
| Fail | Sisu |
|
|
|
|---|---|
|
|
|
-| `dev/docker-compose.yml` | `clickstack`, `clickstack-proxy`, `node-exporter`, `cadvisor`, `otel-collector` (always-on) |
|
|
|
-| `dev/config/otel-collector/config.yaml` | prometheus receiver (node/cadvisor/llamacpp) + regexp-filter + OTLP eksport |
|
|
|
-| `dev/clickstack-entry.sh` | mongod + **OTel-kollektor** + HyperDX app |
|
|
|
-| `dev/.env` | `HYPERDX_INGEST_KEY` (gitignore'itud) — kollektori OTLP auth |
|
|
|
+| `dev/docker-compose.yml` | `clickstack` (app), `clickstack-db` (mongo), `otel-collector`, `clickstack-proxy`, `node-exporter`, `cadvisor`, `otel-ratelimit`, `alert-relay` |
|
|
|
+| `dev/config/otel-collector/custom.config.yaml` | prometheus (node/cadvisor/llamacpp) + filelog (konteinerilogid) + filtrid; pipeline'id → base `clickhouse` |
|
|
|
+| `dev/config/clickstack-proxy/nginx.conf` | 8124 proxy (`/clickstack/`→302, `/api/`→app, `/`→ClickHouse) |
|
|
|
+| `dev/config/otel-ratelimit/nginx.conf` | rate-limit `otel.odamus.com` → `otel-collector:4318` |
|
|
|
+| `dev/config/alert-relay/alert_relay.py` | HyperDX webhook → Telegram |
|
|
|
+| `dev/.env` | `HYPERDX_INGEST_KEY`, `TELEGRAM_BOT_TOKEN` jm (gitignore'itud) |
|
|
|
|
|
|
### ClickStack env (olulised)
|
|
|
-- `CLICKHOUSE_ENDPOINT=tcp://clickhouse:9000?dial_timeout=10s`
|
|
|
-- `CLICKHOUSE_USER=default`, `CLICKHOUSE_PASSWORD=dev_clickhouse_2026`
|
|
|
-- `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=default`
|
|
|
+- App: `MONGO_URI=mongodb://clickstack-db:27017/hyperdx`, `HYPERDX_API_KEY=...`, `OPAMP_PORT=4320`,
|
|
|
+ `OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318`,
|
|
|
+ `NEXT_PUBLIC_OTEL_EXPORTER_OTLP_ENDPOINT=http://100.87.1.24:4318` (brauser / session replay).
|
|
|
+- Collector: `CLICKHOUSE_ENDPOINT=tcp://clickhouse:9000?dial_timeout=10s`, `CLICKHOUSE_USER/PASSWORD`,
|
|
|
+ `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=default`, `OPAMP_SERVER_URL=http://clickstack:4320`,
|
|
|
+ `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml`.
|
|
|
- `DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://clickhouse:8123","username":"default","password":"dev_clickhouse_2026"}]`
|
|
|
|
|
|
## ClickHouse ühendus (HyperDX Connections) — ÕIGE seadistus
|
