# Observability (ClickStack / HyperDX) — ohmu

**Loodud 2026-09-25.** ClickStack = HyperDX UI + (väline) ClickHouse + kaasas olev
OTel-kollektor. Kogu grupp on **always-on** (ei ole enam `experimental`).

## Ligipääs
- UI: **http://100.87.1.24:8124/clickstack/search** (läbi `clickstack-proxy`)
- UI otse: http://100.87.1.24:18080
- Kasutaja: `admin@odamus.com` / `Dev_clickhouse_2026!`
- ClickHouse (väline, `clickhouse` konteiner): HTTP 8125, native 9004 (Tailscale-only)

## Arhitektuur (andmevoog)
```
node-exporter ─┐
cadvisor ──────┼─► otel-collector (scrape 60s) ─► clickstack:4317 (kaasas kollektor) ─► ClickHouse (default.otel_*)
llama.cpp ─────┘        (main_ai_network)              (opampsupervisor)
```
- `otel-collector` (contrib) skrape'ib node-exporter, cadvisor ja **llama.cpp** (`host.docker.internal:8070/metrics`).
- Kaasas olev kollektor (clickstack konteineris) võtab configu API-lt (opamp, port 4320) ja kirjutab ClickHouse'i `otel_*` tabelitesse.
- **Kriitiline:** `clickstack-entry.sh` peab käivitama `/otel-entrypoint.sh /usr/local/bin/opampsupervisor` — muidu OTLP andmed ei püsi.

## ClickHouse tabelid (default)
`otel_logs`, `otel_traces`, `otel_metrics_gauge|sum|histogram|exponential_histogram|summary` + rollupid.
Skeem luuakse automaatselt schema-seed'iga (goose).

## Konfiguratsioon
| Fail | Sisu |
|---|---|
| `dev/docker-compose.yml` | `clickstack`, `clickstack-proxy`, `node-exporter`, `cadvisor`, `otel-collector` (always-on) |
| `dev/config/otel-collector/config.yaml` | prometheus receiver (node/cadvisor/llamacpp) + regexp-filter + OTLP eksport |
| `dev/clickstack-entry.sh` | mongod + **OTel-kollektor** + HyperDX app |
| `dev/.env` | `HYPERDX_INGEST_KEY` (gitignore'itud) — kollektori OTLP auth |

### ClickStack env (olulised)
- `CLICKHOUSE_ENDPOINT=tcp://clickhouse:9000?dial_timeout=10s`
- `CLICKHOUSE_USER=default`, `CLICKHOUSE_PASSWORD=dev_clickhouse_2026`
- `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=default`
- `DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://default:dev_clickhouse_2026@clickhouse:8123"}]`

## Teadaolev lõks: „X-ClickHouse HTTP headers and Authorization simultaneously"
HyperDX esiosa saadab CH-proxy'le `Authorization` (Bearer) ja proxy lisab `X-ClickHouse-User/Key`.
ClickHouse keeldub mõlemast korraga (Code 516). **Lahendus:** hoia ühenduse
`username`/`password` **tühjad** ja pane tunnused **host-URL-i**
(`http://default:parool@clickhouse:8123`) — siis läheb ainult `Authorization: Basic`.
(Diagnoos: `curl -H 'X-ClickHouse-User: default' -H 'X-ClickHouse-Key: ...' -H 'Authorization: Bearer x' 'http://100.87.1.24:8125/?query=SELECT%201'`.)

## Metrikad (praegu)
- **Host:** RAM (available/total), swap, load, CPU, failisüsteem, võrk (node-exporter).
- **Konteinerid:** CPU, mälu (working set/usage), võrk (cadvisor).
- **Mudel:** llama.cpp — prompt/ predicted tokens, requests_processing, kv_cache (`--metrics`).

## Dashboardid (retseptid)
HyperDX UI-s: **Dashboards → New dashboard → Add tile**. Kasuta allolevaid metrikaid
(Metrics source) või "SQL" diagrammi. Ajatelje jaoks sobib `TimeUnix`.

| Paneel | Mõõdik / SQL |
|---|---|
| RAM vaba | `node_memory_MemAvailable_bytes` (gauge) |
| Swap vaba | `node_memory_SwapFree_bytes` (gauge) |
| Load | `node_load1` (gauge) |
| Konteineri mälu | `container_memory_working_set_bytes` (group: `Attributes['name']`) |
| Konteineri CPU | `container_cpu_usage_seconds_total` (rate) |
| **Llama tokeneid/s** | `rate(llamacpp:tokens_predicted_total)` või pika aja trend |
| **Llama päringud** | `llamacpp:requests_processing` (gauge), `llamacpp:tokens_predicted_total` (sum) |
| **Llama KV-cache** | `llamacpp:kv_cache_usage_ratio` (gauge) |
| **Teenus üleval** | `ohmu_service_up` (group: `Attributes['service']`) |
| **Restardid** | `ohmu_container_restarts` (group: `Attributes['name']`) |

Näidis-SQL (ClickHouse):
```sql
-- Teenuste seis viimase 10 min jooksul
SELECT toStartOfMinute(TimeUnix) AS t, Attributes['service'] AS service, avg(Value) AS up
FROM default.otel_metrics_gauge
WHERE MetricName = 'ohmu_service_up' AND TimeUnix > now() - INTERVAL 10 MINUTE
GROUP BY t, service ORDER BY t;
```

## Seotud
- Vahejuhtumite audit: `../LOEMIND_seisevad_konteinerid.md`
- Serveri mudelid: `~/servers/LOEMIND/llama-server.md`