observability: ClickStack always-on + node-exporter/cadvisor/otel-collector; otel_* metrikad -> ClickHouse; auth-fix (URL creds)

dev/LOEMIND_observability.md

@@ -0,0 +1,54 @@
+# Observability (ClickStack / HyperDX) — ohmu
+
+**Loodud 2026-09-25.** ClickStack = HyperDX UI + (väline) ClickHouse + kaasas olev
+OTel-kollektor. Kogu grupp on **always-on** (ei ole enam `experimental`).
+
+## Ligipääs
+- UI: **http://100.87.1.24:8124/clickstack/search** (läbi `clickstack-proxy`)
+- UI otse: http://100.87.1.24:18080
+- Kasutaja: `admin@odamus.com` / `Dev_clickhouse_2026!`
+- ClickHouse (väline, `clickhouse` konteiner): HTTP 8125, native 9004 (Tailscale-only)
+
+## Arhitektuur (andmevoog)
+```
+node-exporter ─┐
+cadvisor ──────┼─► otel-collector (scrape 60s) ─► clickstack:4317 (kaasas kollektor) ─► ClickHouse (default.otel_*)
+llama.cpp ─────┘        (main_ai_network)              (opampsupervisor)
+```
+- `otel-collector` (contrib) skrape'ib node-exporter, cadvisor ja **llama.cpp** (`host.docker.internal:8070/metrics`).
+- Kaasas olev kollektor (clickstack konteineris) võtab configu API-lt (opamp, port 4320) ja kirjutab ClickHouse'i `otel_*` tabelitesse.
+- **Kriitiline:** `clickstack-entry.sh` peab käivitama `/otel-entrypoint.sh /usr/local/bin/opampsupervisor` — muidu OTLP andmed ei püsi.
+
+## ClickHouse tabelid (default)
+`otel_logs`, `otel_traces`, `otel_metrics_gauge|sum|histogram|exponential_histogram|summary` + rollupid.
+Skeem luuakse automaatselt schema-seed'iga (goose).
+
+## Konfiguratsioon
+| Fail | Sisu |
+|---|---|
+| `dev/docker-compose.yml` | `clickstack`, `clickstack-proxy`, `node-exporter`, `cadvisor`, `otel-collector` (always-on) |
+| `dev/config/otel-collector/config.yaml` | prometheus receiver (node/cadvisor/llamacpp) + regexp-filter + OTLP eksport |
+| `dev/clickstack-entry.sh` | mongod + **OTel-kollektor** + HyperDX app |
+| `dev/.env` | `HYPERDX_INGEST_KEY` (gitignore'itud) — kollektori OTLP auth |
+
+### ClickStack env (olulised)
+- `CLICKHOUSE_ENDPOINT=tcp://clickhouse:9000?dial_timeout=10s`
+- `CLICKHOUSE_USER=default`, `CLICKHOUSE_PASSWORD=dev_clickhouse_2026`
+- `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=default`
+- `DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://default:dev_clickhouse_2026@clickhouse:8123"}]`
+
+## Teadaolev lõks: „X-ClickHouse HTTP headers and Authorization simultaneously"
+HyperDX esiosa saadab CH-proxy'le `Authorization` (Bearer) ja proxy lisab `X-ClickHouse-User/Key`.
+ClickHouse keeldub mõlemast korraga (Code 516). **Lahendus:** hoia ühenduse
+`username`/`password` **tühjad** ja pane tunnused **host-URL-i**
+(`http://default:parool@clickhouse:8123`) — siis läheb ainult `Authorization: Basic`.
+(Diagnoos: `curl -H 'X-ClickHouse-User: default' -H 'X-ClickHouse-Key: ...' -H 'Authorization: Bearer x' 'http://100.87.1.24:8125/?query=SELECT%201'`.)
+
+## Metrikad (praegu)
+- **Host:** RAM (available/total), swap, load, CPU, failisüsteem, võrk (node-exporter).
+- **Konteinerid:** CPU, mälu (working set/usage), võrk (cadvisor).
+- **Mudel:** llama.cpp — prompt/ predicted tokens, requests_processing, kv_cache (`--metrics`).
+
+## Seotud
+- Vahejuhtumite audit: `../LOEMIND_seisevad_konteinerid.md`
+- Serveri mudelid: `~/servers/LOEMIND/llama-server.md`

dev/clickstack-entry.sh

@@ -3,6 +3,9 @@
 set -e
 cd /app
 mongod --dbpath /data/db --bind_ip 127.0.0.1 --fork --logpath /var/log/mongod.log
+# OTel kollektor (opampsupervisor): võtab API-lt konfiguratsiooni ja haldab otel_* tabeleid ClickHouse'is.
+# Ilma selleta ei püsi OTLP meetrikad/logid/trace'id (ainult API võtab vastu ja viskab ära).
+/otel-entrypoint.sh /usr/local/bin/opampsupervisor > /var/log/otel-collector.log 2>&1 &
 exec node ./node_modules/.bin/concurrently --kill-others-on-fail --names=API,APP,ALERT-TASK \
   "PORT=8000 HYPERDX_APP_PORT=8080 ./packages/api/bin/hyperdx api" \
   "cd ./packages/app/packages/app && HOSTNAME='0.0.0.0' HYPERDX_API_PORT=8000 PORT=8080 node server.js" \

dev/config/otel-collector/config.yaml

@@ -0,0 +1,49 @@
+receivers:
+  prometheus:
+    config:
+      global:
+        scrape_interval: 60s
+      scrape_configs:
+        - job_name: node
+          static_configs:
+            - targets: ["node-exporter:9100"]
+        - job_name: cadvisor
+          static_configs:
+            - targets: ["cadvisor:8080"]
+        - job_name: llamacpp
+          static_configs:
+            - targets: ["host.docker.internal:8070"]
+
+processors:
+  memory_limiter:
+    check_interval: 5s
+    limit_mib: 400
+    spike_limit_mib: 100
+  # Hoia ainult vajalikud metrikad (cadvisor vaikimisi maht > 4 MB -> HyperDX lükkab tagasi)
+  filter/keep:
+    metrics:
+      include:
+        match_type: regexp
+        metric_names:
+          - "^node_(memory_MemAvailable_bytes|memory_MemTotal_bytes|memory_SwapFree_bytes|memory_SwapTotal_bytes|load1|cpu_seconds_total|filesystem_avail_bytes|filesystem_size_bytes|network_receive_bytes_total|network_transmit_bytes_total)$"
+          - "^container_(cpu_usage_seconds_total|memory_working_set_bytes|memory_usage_bytes|network_receive_bytes_total|network_transmit_bytes_total)$"
+          - ".*llamacpp.*"
+  batch:
+    send_batch_size: 500
+    timeout: 10s
+
+exporters:
+  otlp:
+    endpoint: clickstack:4317
+    tls:
+      insecure: true
+    headers:
+      authorization: ${env:HYPERDX_INGEST_KEY}
+    compression: none
+
+service:
+  pipelines:
+    metrics:
+      receivers: [prometheus]
+      processors: [memory_limiter, filter/keep, batch]
+      exporters: [otlp]

dev/docker-compose.yml

@@ -76,7 +76,7 @@ services:
     build: ./nutikodu-web
     container_name: nutikodu-web
     restart: unless-stopped
-    mem_limit: 192m
+    mem_limit: 512m
     env_file:
       - .env
     environment:
@@ -98,7 +98,6 @@ services:
   clickstack:
     image: docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest
     container_name: clickstack
-    profiles: ["experimental"]
     restart: unless-stopped
     environment:
       - TZ=Europe/Tallinn
@@ -110,6 +109,11 @@ services:
       - HYPERDX_API_KEY=790900ff1b76521b3626450576b78715
       - NEXT_PUBLIC_IS_LOCAL_MODE=true
       - EXPRESS_SESSION_SECRET=dev_session_secret_2026
+      - CLICKHOUSE_ENDPOINT=tcp://clickhouse:9000?dial_timeout=10s
+      - CLICKHOUSE_USER=default
+      - CLICKHOUSE_PASSWORD=dev_clickhouse_2026
+      - HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=default
+      - 'DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://default:dev_clickhouse_2026@clickhouse:8123"}]'
     command: ["sh", "/clickstack-entry.sh"]
     ports:
       - "18080:8080"   # HyperDX UI (search, dashboards)
@@ -126,7 +130,6 @@ services:
   clickstack-proxy:
     image: nginx:alpine
     container_name: clickstack-proxy
-    profiles: ["experimental"]
     restart: unless-stopped
     ports:
       - "8124:8124"
@@ -138,6 +141,64 @@ services:
     networks:
       - main_ai_network
 
+  # Hosti metrikad (node-exporter) — Prometheus endpoint :9100
+  node-exporter:
+    image: prom/node-exporter:latest
+    container_name: node-exporter
+    restart: unless-stopped
+    mem_limit: 128m
+    command:
+      - "--path.rootfs=/host"
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/host:ro,rslave
+    networks:
+      - main_ai_network
+
+  # Konteinerite metrikad (cadvisor) — Prometheus endpoint :8080
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:latest
+    container_name: cadvisor
+    restart: unless-stopped
+    mem_limit: 256m
+    privileged: true
+    command:
+      - "--docker_only=true"
+      - "--disable_metrics=advtcp,cpu_topology,cpuset,hugetlb,memory_numa,process,referenced_memory,resctrl,sched,tcp,udp,percpu"
+    devices:
+      - /dev/kmsg
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+      - /dev/disk/:/dev/disk:ro
+    networks:
+      - main_ai_network
+
+  # OTel kollektor: scrape node-exporter + cadvisor -> OTLP -> clickstack
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    container_name: otel-collector
+    restart: unless-stopped
+    mem_limit: 512m
+    command: ["--config=/etc/otel/config.yaml"]
+    environment:
+      - HYPERDX_INGEST_KEY=${HYPERDX_INGEST_KEY}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - ./config/otel-collector/config.yaml:/etc/otel/config.yaml:ro
+    depends_on:
+      - clickstack
+      - node-exporter
+      - cadvisor
+    networks:
+      - main_ai_network
+
 networks:
   main_ai_network:
     external: true