observability: filelog logide pipeline + auth fix (eraldi username/password) + metric select schema + alert number-paneel

dev/LOEMIND_observability.md

@@ -35,14 +35,37 @@ Skeem luuakse automaatselt schema-seed'iga (goose).
 - `CLICKHOUSE_ENDPOINT=tcp://clickhouse:9000?dial_timeout=10s`
 - `CLICKHOUSE_USER=default`, `CLICKHOUSE_PASSWORD=dev_clickhouse_2026`
 - `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=default`
-- `DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://default:dev_clickhouse_2026@clickhouse:8123"}]`
+- `DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://clickhouse:8123","username":"default","password":"dev_clickhouse_2026"}]`
 
-## Teadaolev lõks: „X-ClickHouse HTTP headers and Authorization simultaneously"
-HyperDX esiosa saadab CH-proxy'le `Authorization` (Bearer) ja proxy lisab `X-ClickHouse-User/Key`.
-ClickHouse keeldub mõlemast korraga (Code 516). **Lahendus:** hoia ühenduse
-`username`/`password` **tühjad** ja pane tunnused **host-URL-i**
-(`http://default:parool@clickhouse:8123`) — siis läheb ainult `Authorization: Basic`.
-(Diagnoos: `curl -H 'X-ClickHouse-User: default' -H 'X-ClickHouse-Key: ...' -H 'Authorization: Bearer x' 'http://100.87.1.24:8125/?query=SELECT%201'`.)
+## ClickHouse ühendus (HyperDX Connections) — ÕIGE seadistus
+Ühendusel peavad olema **eraldi väljad** `username`/`password` ja `host` **ilma
+userinfo'ta** (`http://clickhouse:8123`). Siis lisab CH-proxy `X-ClickHouse-User`/`Key` päised.
+
+> ❗ Varasem soovitus hoida `username`/`password` tühjad ja panna tunnused
+> host-URL-i (`http://default:parool@clickhouse:8123`) **ei tööta**: `http-proxy-middleware`
+> ei edasta host-URL-i parooli → CH saab kasutaja `default` tühja parooliga →
+> „Authentication failed: password is incorrect". (Kinnitatud `tcpdump -i any 'tcp port 8123'`-ga.)
+
+Mongo otsene parandus:
+```javascript
+db.connections.updateOne({_id: ObjectId("<conn-id>")},
+  {$set:{host:"http://clickhouse:8123", username:"default", password:"dev_clickhouse_2026"}});
+```
+> Pärast otsest mongo-muudatust **taaskäivita `clickstack`** (muidu jääb vahemällu):
+> `docker restart clickstack`.
+
+**Code 516 hoiatus:** kui ühendusel on `username` JA päring sisaldab `user=` query-param,
+tuleb `Code: 516 ... X-ClickHouse HTTP headers and authentication via parameters simultaneously`.
+Seetõttu ei tohi ühendust anda browseri CH-kliendile koos username'iga; HyperDX metadataklient
+(`W({queryTimeout})`) ehitab kliendi **ilma** username'ita → OK.
+
+## Logide kogumine (filelog)
+`otel-collector` loeb Dockeri konteineriloge:
+- `receivers.filelog`: `include: /var/lib/docker/containers/*/*-json.log`, `container` parser, `start_at: end`
+- `service.pipelines.logs`: `[filelog] → [memory_limiter, batch] → [otlp]`
+- Compose: `otel-collector` vajab **mounti** `/var/lib/docker/containers:/var/lib/docker/containers:ro`
+  **ja `user: "0:0"`** (mitte-root kasutaja → „open .: permission denied").
+- Kontroll: `SELECT count(), max(Timestamp) FROM default.otel_logs`.
 
 ## Metrikad (praegu)
 - **Host:** RAM (available/total), swap, load, CPU, failisüsteem, võrk (node-exporter).
@@ -77,6 +100,21 @@ GROUP BY t, service ORDER BY t;
 
 ## Teadaolevad UI-nõksud
 
+### Metrika-paneeli `select` skeem (HyperDX 2.37)
+Metrika-diagramm ja alert nõuavad, et `select[]` elemendil oleks:
+`{metricName, metricType, aggFn, isDelta, filters: [], valueExpression: "Value"}`,
+kus `metricType` ∈ `gauge|sum|histogram|summary|exponential histogram`.
+Vana `aggCondition`-ainult kuju annab **`no query support for metric type=undefined`**.
+`valueExpression` on vajalik, sest alerti config lisab `variables` → macro-töötlus loeb
+`select[].valueExpression` (puudumisel `TypeError: Cannot read properties of undefined (reading 'length')`).
+`metricType` määra ClickHouse'i tabeli järgi (`otel_metrics_gauge|sum|histogram|...`):
+nt `llamacpp:tokens_predicted_total` → `sum`, `node_load1`/`ohmu_service_up` → `gauge`.
+
+### Alert nõuab number/line/stacked bar paneeli
+Alert **ei toeta `table`-paneeli** (`Unsupported alert source: tile`). Kasuta
+`number`/`line`/`stacked bar` paneeli. Pärast alerti/paneeli otsest mongo-muudatust
+**taaskäivita `clickstack`** (vahemälu).
+
 ### Heatmap: „Value expression is required for heatmap charts"
 Chart Explorer → Heatmap nõuab, et seeria **Value expression** oleks täidetud
 (HyperDX valideerib `series[0].valueExpression`). See on UI-konfiguratsioon, mitte andmeviga.

dev/config/otel-collector/config.yaml

@@ -14,6 +14,19 @@ receivers:
           static_configs:
             - targets: ["host.docker.internal:8070"]
 
+  # Konteinerite logid: Docker JSON-logid otse /var/lib/docker/containers/*.
+  # `container` parser võtab log keha + lisab container.name/id ja log.iostream.
+  # start_at: end -> ära loe ajalugu (ainult uued read alates käivitamisest).
+  filelog:
+    include:
+      - /var/lib/docker/containers/*/*-json.log
+    include_file_path: true
+    start_at: end
+    operators:
+      - type: container
+        id: container-parser
+        format: docker
+
 processors:
   memory_limiter:
     check_interval: 5s
@@ -47,3 +60,7 @@ service:
       receivers: [prometheus]
       processors: [memory_limiter, filter/keep, batch]
       exporters: [otlp]
+    logs:
+      receivers: [filelog]
+      processors: [memory_limiter, batch]
+      exporters: [otlp]

dev/docker-compose.yml

@@ -113,7 +113,7 @@ services:
       - CLICKHOUSE_USER=default
       - CLICKHOUSE_PASSWORD=dev_clickhouse_2026
       - HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=default
-      - 'DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://default:dev_clickhouse_2026@clickhouse:8123"}]'
+      - 'DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://clickhouse:8123","username":"default","password":"dev_clickhouse_2026"}]'
     command: ["sh", "/clickstack-entry.sh"]
     ports:
       - "127.0.0.1:18080:8080"
@@ -189,6 +189,7 @@ services:
     container_name: otel-collector
     restart: unless-stopped
     mem_limit: 512m
+    user: "0:0"   # root: vaja lugeda /var/lib/docker/containers/*-json.log (710 root:root)
     command: ["--config=/etc/otel/config.yaml"]
     environment:
       - HYPERDX_INGEST_KEY=${HYPERDX_INGEST_KEY}
@@ -196,6 +197,7 @@ services:
       - "host.docker.internal:host-gateway"
     volumes:
       - ./config/otel-collector/config.yaml:/etc/otel/config.yaml:ro
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
     depends_on:
       - clickstack
       - node-exporter