varvusensor: AS7341 varvisensor + avalik MQTT broker (edge), CH parandused

- uus dev/varvusensor: ESPHome config (AS7341 bait-vahetuse fix, mitu wifi vorgu, MQTT TLS), koguja MQTT->ClickHouse, color/calibrate/live tooRiistad

- avalik MQTT edge-l (mqtt.odamus.com:8883 TLS) — ESP32 ei toeta Tailscale-i, seega universaalne igast vorgust

- CH: network_compression_method=LZ4 (users.d) — parandab DBeaveri Invalid LZ4 magic byte

- CH: HTTP/native ka LAN-IP-l (192.168.68.114)

dev/LOEMIND.md

@@ -9,19 +9,27 @@ mille eesmärk on Oracle-baaside ülemineku ettevalmistamine).
 
 | Omadus | Väärtus |
 |--------|---------|
-| Pilt | `clickhouse/clickhouse-server:26.8.1.2041` |
+| Pilt | `clickhouse/clickhouse-server:26.9.14.10` |
 | Konteiner | `clickhouse` |
 | Püsivus | **alati aktiivne** (ei ole enam `experimental` profiilis), `restart: unless-stopped`, `mem_limit: 2g` |
-| HTTP API | **8125**, seotud **ainult Tailscale'ile** (`100.87.1.24:8125`) — **8124** läbi `clickstack-proxy` |
-| Native TCP | **9004**, seotud ainult Tailscale'ile (`100.87.1.24:9004`) |
+| HTTP API | **8125** — `100.87.1.24:8125` (Tailscale) + `192.168.68.114:8125` (LAN); **8124** läbi `clickstack-proxy` |
+| Native TCP | **9004** — `100.87.1.24:9004` (Tailscale) + `192.168.68.114:9004` (LAN) |
 | Kasutaja | `default` (ja `dev`, `zigbee`) |
 | Parool | `dev_clickhouse_2026` (default/dev); `zigbee` kasutaja parool on `dev/.env` failis |
 | Andmed | `./data/clickhouse` (gitignore'd) |
-| Config | `./config/` (merge'd `/etc/clickhouse-server/config.d`) |
+| Config | `./config/` → `config.d` (server); `./users.d/` → `users.d` (users + **profiilid**) |
 | Võrk | `main_ai_network` |
 
-> **Turvamärkus (2026-09-17):** pordid 8125/9004 ei ole enam `0.0.0.0` peal, vaid
-> ainult Tailscale'i aadressil `100.87.1.24`. LAN-ist ligi ei pääse.
+> **Turvamärkus (2026-09-17, uuendatud 2026-10-17):** pordid 8125/9004 ei ole
+> `0.0.0.0` peal. Algselt ainult Tailscale'i aadressil `100.87.1.24`; 2026-10-17
+> lisati ka **LAN-IP `192.168.68.114`** (DBeaver). Auth on kohustuslik.
+
+> **LZ4/ZSTD (2026-10-17):** ClickHouse 26.x vaikimisi
+> `network_compression_method=ZSTD` murdis vanemad clickhouse-jdbc/DBeaver
+> kliendid → *"Invalid LZ4 magic byte"* (bait `0x90` = ZSTD, klient ootab LZ4
+> `0x82`). Lahendus: `./users.d/zz-network-compression.xml` seab profiilis
+> `default` → `network_compression_method=lz4`. **NB:** profiilid kuuluvad
+> `users.d`-sse, MITTE `config.d`-sse (config.xml-s pole `<profiles>`).
 
 ## Web UI
 

dev/docker-compose.yml

@@ -13,9 +13,12 @@ services:
     ports:
       - "100.87.1.24:8125:8123"   # ClickHouse HTTP (ainult Tailscale)
       - "100.87.1.24:9004:9000"   # native TCP (ainult Tailscale)
+      - "192.168.68.114:8125:8123"   # ClickHouse HTTP (LAN, DBeaver)
+      - "192.168.68.114:9004:9000"   # native TCP (LAN)
     volumes:
       - ./data/clickhouse:/var/lib/clickhouse
       - ./config:/etc/clickhouse-server/config.d
+      - ./users.d/zz-network-compression.xml:/etc/clickhouse-server/users.d/zz-network-compression.xml:ro
       - /etc/localtime:/etc/localtime:ro
       - /usr/share/zoneinfo:/usr/share/zoneinfo:ro
     ulimits:

dev/users.d/zz-network-compression.xml

@@ -0,0 +1,11 @@
+<?xml version="1.0"?>
+<clickhouse>
+    <profiles>
+        <default>
+            <!-- ClickHouse 26.x vaikimisi network_compression_method=ZSTD
+                 murdis vanemad clickhouse-jdbc / DBeaver klientid (eeldavad
+                 LZ4 -> "Invalid LZ4 magic byte"). Tagasi klassikaliseks LZ4-ks. -->
+            <network_compression_method>lz4</network_compression_method>
+        </default>
+    </profiles>
+</clickhouse>

dev/varvusensor/.gitignore

@@ -0,0 +1,5 @@
+# ESPHome arenduskeskkonna jäägid
+.venv/
+compile.log
+# mosquitto paroolihash (loodud eraldi, vt LOEMIND)
+mosquitto/passwd

dev/varvusensor/collector/Dockerfile

@@ -0,0 +1,10 @@
+FROM python:3.12-slim
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY collector.py color.py ./
+
+CMD ["python", "-u", "collector.py"]

dev/varvusensor/collector/calibrate.py

@@ -0,0 +1,128 @@
+"""Kalibreerimise abivahend AS7341 värviandurile.
+
+Loeb ClickHouse'ist (vaikimisi uus67.varvus) ja toetab:
+  * capture-white: salvesta valge referents (viimased N mõõtmist) -> calibration.json
+  * show:          näita viimaseid mõõtmisi (toores hex vs kalibreeritud hex)
+
+Kalibreeritud värvus = iga kanal jagatud valge referentsiga (reflectance),
+mis eemaldab valgustuse värvitooni ja anduri kanalite erinevuse.
+
+Käivitamine (projekti .venv):
+  CH_URL=http://100.87.1.24:8125 CH_USER=zigbee CH_PASS=... \
+    .venv/bin/python varvusensor/collector/calibrate.py capture-white --last 5
+  CH_URL=http://100.87.1.24:8125 CH_USER=zigbee CH_PASS=... \
+    .venv/bin/python varvusensor/collector/calibrate.py show --last 10
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import statistics
+import sys
+from pathlib import Path
+
+import requests
+
+import color
+from settings import CH_URL, CH_DB, CH_TABLE, CH_USER, CH_PASS
+
+CALIB_FILE = Path(
+    os.environ.get("CALIB_FILE", Path(__file__).with_name("calibration.json"))
+)
+
+FIELDS = ("f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "clear", "nir")
+BANDS = tuple(color.CHANNEL_WAVELENGTHS)  # f1..f8
+
+
+def ch_query(sql: str) -> str:
+    resp = requests.post(
+        f"{CH_URL}/",
+        params={"database": CH_DB, "query": sql},
+        auth=(CH_USER, CH_PASS),
+        timeout=15,
+    )
+    resp.raise_for_status()
+    return resp.text
+
+
+def fetch_rows(last: int) -> list[dict]:
+    cols = ", ".join(FIELDS + ("rgb_hex",))
+    sql = (
+        f"SELECT ts, {cols} FROM {CH_DB}.{CH_TABLE} "
+        f"ORDER BY ts DESC LIMIT {int(last)} FORMAT JSONEachRow"
+    )
+    return [json.loads(line) for line in ch_query(sql).strip().splitlines() if line]
+
+
+def capture_white(last: int) -> int:
+    rows = fetch_rows(last)
+    if not rows:
+        print("Andmeid pole — mõõda esmalt valget pinda.")
+        return 1
+    ref: dict[str, float] = {}
+    for field in FIELDS:
+        values = [r[field] for r in rows if r.get(field) is not None]
+        ref[field] = round(statistics.mean(values), 3) if values else 0.0
+    payload = {"reference_rows": len(rows), "white": ref}
+    CALIB_FILE.write_text(json.dumps(payload, indent=2, ensure_ascii=False))
+    print(f"Valge referents salvestatud ({len(rows)} reast) -> {CALIB_FILE}")
+    print(json.dumps(ref, indent=2))
+    return 0
+
+
+def load_white() -> dict[str, float] | None:
+    if not CALIB_FILE.exists():
+        return None
+    return json.loads(CALIB_FILE.read_text()).get("white")
+
+
+def reflectance(channels: dict[str, float], white: dict[str, float]) -> dict[str, float]:
+    out: dict[str, float] = {}
+    for band in BANDS:
+        ref = white.get(band, 0.0)
+        value = channels.get(band)
+        if value is not None and ref:
+            out[band] = value / ref
+    return out
+
+
+def show(last: int) -> int:
+    rows = fetch_rows(last)
+    white = load_white()
+    if white is None:
+        print("(kalibratsioon puudub — näitan ainult toorest; tee capture-white)")
+    for row in rows:
+        channels = {b: row[b] for b in BANDS if row.get(b) is not None}
+        raw = color.analyze(channels)
+        line = (
+            f"{row['ts']}  toores {raw['rgb_hex']} "
+            f"(x={raw['x_chrom']:.3f} y={raw['y_chrom']:.3f})"
+        )
+        if white is not None:
+            cal = color.analyze(reflectance(channels, white))
+            line += (
+                f"  | kalib. {cal['rgb_hex']} "
+                f"(x={cal['x_chrom']:.3f} y={cal['y_chrom']:.3f})"
+            )
+        print(line)
+    return 0
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="AS7341 kalibreerimine")
+    sub = parser.add_subparsers(dest="cmd", required=True)
+    cw = sub.add_parser("capture-white", help="salvesta valge referents")
+    cw.add_argument("--last", type=int, default=5)
+    sh = sub.add_parser("show", help="näita viimaseid mõõtmisi")
+    sh.add_argument("--last", type=int, default=10)
+    args = parser.parse_args()
+    if args.cmd == "capture-white":
+        return capture_white(args.last)
+    if args.cmd == "show":
+        return show(args.last)
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

dev/varvusensor/collector/collector.py

@@ -0,0 +1,106 @@
+"""AS7341 värvianduri koguja: MQTT -> ClickHouse (uus67.varvus).
+
+Tellib MQTT-teemat (vaikimisi varvus/spectral), kus ESPHome saadab
+JSON-i kanalitega f1..f8, clear, nir (+ gain). Iga sonum = uks rida.
+Lisaks arvutatakse ligikaudne värvus (CIE XYZ -> sRGB, vt color.py).
+
+Tellimus tehakse on_connect-is, et see taastuks ka parast brokeri
+taaskäivitust / ühenduse katkemist (clean session).
+"""
+from __future__ import annotations
+
+import json
+import os
+from datetime import datetime
+from zoneinfo import ZoneInfo
+
+import paho.mqtt.client as mqtt
+import requests
+
+import color
+
+MQTT_HOST = os.environ.get("MQTT_HOST", "mosquitto")
+MQTT_PORT = int(os.environ.get("MQTT_PORT", "1883"))
+MQTT_USER = os.environ.get("MQTT_USER", "")
+MQTT_PASS = os.environ.get("MQTT_PASS", "")
+MQTT_TOPIC = os.environ.get("MQTT_TOPIC", "varvus/spectral")
+MQTT_TLS = os.environ.get("MQTT_TLS", "false").strip().lower() in ("1", "true", "yes", "on")
+
+CH_URL = os.environ.get("CH_URL", "http://clickhouse:8123").rstrip("/")
+CH_DB = os.environ.get("CH_DB", "uus67")
+CH_TABLE = os.environ.get("CH_TABLE", "varvus")
+CH_USER = os.environ.get("CH_USER", "zigbee")
+CH_PASS = os.environ.get("CH_PASS", "")
+
+TZ = ZoneInfo("Europe/Tallinn")
+FIELDS = ("f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "clear", "nir")
+
+
+def insert(row: dict) -> None:
+    resp = requests.post(
+        f"{CH_URL}/",
+        params={
+            "database": CH_DB,
+            "query": f"INSERT INTO {CH_DB}.{CH_TABLE} FORMAT JSONEachRow",
+        },
+        data=json.dumps(row, ensure_ascii=False) + "\n",
+        auth=(CH_USER, CH_PASS),
+        timeout=10,
+    )
+    resp.raise_for_status()
+
+
+def on_connect(client, userdata, flags, reason_code, properties=None) -> None:
+    if reason_code == 0:
+        client.subscribe(MQTT_TOPIC, qos=0)
+        print(f"MQTT ühendatud, tellitud {MQTT_TOPIC}", flush=True)
+    else:
+        print(f"MQTT ühenduse viga: {reason_code}", flush=True)
+
+
+def on_message(client, userdata, msg) -> None:
+    try:
+        payload = json.loads(msg.payload.decode())
+    except Exception as exc:
+        print(f"JSON viga: {exc}", flush=True)
+        return
+    row: dict = {
+        "ts": datetime.now(TZ).strftime("%Y-%m-%d %H:%M:%S"),
+        "device": "varvusensor",
+    }
+    channels: dict[str, float] = {}
+    for field in FIELDS:
+        value = payload.get(field)
+        if value is not None:
+            channels[field] = float(value)
+            row[field] = float(value)
+    gain = payload.get("gain")
+    if gain is not None:
+        row["gain"] = int(gain)
+    row.update(color.analyze(channels))
+    try:
+        insert(row)
+        print(
+            f"salvestatud: {row['rgb_hex']} x={row['x_chrom']} y={row['y_chrom']} "
+            f"f5={row.get('f5')} f6={row.get('f6')}",
+            flush=True,
+        )
+    except Exception as exc:
+        print(f"ClickHouse viga: {exc}", flush=True)
+
+
+def main() -> None:
+    client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
+    if MQTT_TLS:
+        client.tls_set()
+    if MQTT_USER:
+        client.username_pw_set(MQTT_USER, MQTT_PASS)
+    client.on_connect = on_connect
+    client.on_message = on_message
+    client.connect(MQTT_HOST, MQTT_PORT, 60)
+    print(f"Koguja käivitub, broker {MQTT_HOST}:{MQTT_PORT} (TLS={MQTT_TLS})", flush=True)
+    client.loop_forever()
+
+
+if __name__ == "__main__":
+    main()

dev/varvusensor/collector/color.py

@@ -0,0 +1,101 @@
+"""AS7341 spektri -> ligikaudne värvus (CIE 1931 XYZ -> sRGB).
+
+Kasutab Wyman, Sloan & Shirley (2013) analüütilist CIE 1931 2° CMF
+lähendust 8 kanali kesk lainepikkustel.
+
+NB! See EI ole kalibreeritud kolorimeetria — absoluutse värvuse jaoks on
+vaja valge referentsi ja fikseeritud valgustust (vt LOEMIND.md).
+"""
+from __future__ import annotations
+
+import math
+
+# ESPHome as7341 kanalite F1..F8 kesk lainepikkused (nm)
+CHANNEL_WAVELENGTHS: dict[str, int] = {
+    "f1": 415,
+    "f2": 445,
+    "f3": 480,
+    "f4": 515,
+    "f5": 555,
+    "f6": 590,
+    "f7": 630,
+    "f8": 680,
+}
+
+
+def _piecewise_gaussian(x: float, mu: float, sigma1: float, sigma2: float) -> float:
+    sigma = sigma1 if x < mu else sigma2
+    return math.exp(-0.5 * ((x - mu) / sigma) ** 2)
+
+
+def cie_xyz_cmf(wavelength_nm: float) -> tuple[float, float, float]:
+    """Wyman et al. (2013) lähendus CIE 1931 2° värvusfunktsioonidele."""
+    x_bar = (
+        1.056 * _piecewise_gaussian(wavelength_nm, 599.8, 37.9, 31.0)
+        + 0.362 * _piecewise_gaussian(wavelength_nm, 442.0, 16.0, 26.7)
+        - 0.065 * _piecewise_gaussian(wavelength_nm, 501.1, 20.4, 26.2)
+    )
+    y_bar = (
+        0.821 * _piecewise_gaussian(wavelength_nm, 568.8, 46.9, 40.5)
+        + 0.286 * _piecewise_gaussian(wavelength_nm, 530.9, 16.3, 31.1)
+    )
+    z_bar = (
+        1.217 * _piecewise_gaussian(wavelength_nm, 437.0, 11.8, 36.0)
+        + 0.681 * _piecewise_gaussian(wavelength_nm, 459.0, 26.0, 13.8)
+    )
+    return x_bar, y_bar, z_bar
+
+
+def channels_to_xyz(channels: dict[str, float]) -> tuple[float, float, float]:
+    """Ligikaudne XYZ: iga kanali väärtus kaalutakse CMF-iga kesk lainepikkusel."""
+    x = y = z = 0.0
+    for name, wavelength in CHANNEL_WAVELENGTHS.items():
+        value = channels.get(name)
+        if value is None:
+            continue
+        cx, cy, cz = cie_xyz_cmf(wavelength)
+        x += value * cx
+        y += value * cy
+        z += value * cz
+    return x, y, z
+
+
+def xyz_to_srgb(x: float, y: float, z: float) -> tuple[int, int, int]:
+    """XYZ -> lineaarne sRGB (D65) -> gamma -> 8-bit. Heledus normaliseeritakse."""
+    r_lin = 3.2406 * x - 1.5372 * y - 0.4986 * z
+    g_lin = -0.9689 * x + 1.8758 * y + 0.0415 * z
+    b_lin = 0.0557 * x - 0.2040 * y + 1.0570 * z
+
+    def gamma(c: float) -> float:
+        c = max(0.0, c)
+        return 12.92 * c if c <= 0.0031308 else 1.055 * c ** (1 / 2.4) - 0.055
+
+    r, g, b = gamma(r_lin), gamma(g_lin), gamma(b_lin)
+    scale = max(r, g, b, 1e-9)
+    return (
+        int(round(255 * r / scale)),
+        int(round(255 * g / scale)),
+        int(round(255 * b / scale)),
+    )
+
+
+def chromaticity_xy(x: float, y: float, z: float) -> tuple[float, float]:
+    total = x + y + z
+    if total <= 0:
+        return 0.0, 0.0
+    return x / total, y / total
+
+
+def analyze(channels: dict[str, float]) -> dict:
+    """Tagastab tuletatud värviväljad ClickHouse'i rea jaoks."""
+    x, y, z = channels_to_xyz(channels)
+    cx, cy = chromaticity_xy(x, y, z)
+    r, g, b = xyz_to_srgb(x, y, z)
+    return {
+        "x_chrom": round(cx, 5),
+        "y_chrom": round(cy, 5),
+        "r": r,
+        "g": g,
+        "b": b,
+        "rgb_hex": f"#{r:02X}{g:02X}{b:02X}",
+    }

dev/varvusensor/collector/live.py

@@ -0,0 +1,95 @@
+"""Reaalajas AS7341 värvinäitaja.
+
+Loeb ClickHouse'ist viimase mõõtmise ja kuvab terminalis värvinäidise (24-bitine
+ANSI) + kanalite spektri. Mõeldud katseteks (eri värvi pindade mõõtmine).
+
+Käivitamine (projekti .venv):
+  CH_URL=http://100.87.1.24:8125 CH_USER=zigbee CH_PASS=<parool> \
+    .venv/bin/python varvusensor/collector/live.py [--interval 2] [--once]
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import time
+
+import requests
+
+import color
+from settings import CH_URL, CH_DB, CH_TABLE, CH_USER, CH_PASS
+
+BANDS = color.CHANNEL_WAVELENGTHS  # {f1: 415, ...}
+RESET = "\x1b[0m"
+
+
+def fetch_latest() -> dict | None:
+    cols = ", ".join(BANDS)
+    sql = (
+        f"SELECT ts, {cols}, rgb_hex, x_chrom, y_chrom, r, g, b "
+        f"FROM {CH_DB}.{CH_TABLE} ORDER BY ts DESC LIMIT 1 FORMAT JSONEachRow"
+    )
+    resp = requests.post(
+        f"{CH_URL}/",
+        params={"database": CH_DB, "query": sql},
+        auth=(CH_USER, CH_PASS),
+        timeout=10,
+    )
+    resp.raise_for_status()
+    lines = [line for line in resp.text.strip().splitlines() if line]
+    return json.loads(lines[0]) if lines else None
+
+
+def swatch(r: int, g: int, b: int, width: int = 28, height: int = 3) -> str:
+    block = f"\x1b[48;2;{r};{g};{b}m" + " " * width + RESET
+    return "\n".join([block] * height)
+
+
+def spectrum_bar(channels: dict[str, float]) -> str:
+    vmax = max((channels.get(band) or 0.0) for band in BANDS) or 1.0
+    lines = []
+    for band, wavelength in BANDS.items():
+        value = channels.get(band) or 0.0
+        bars = int(round(40 * value / vmax))
+        lines.append(f"  {band} {wavelength:3d}nm |{'█' * bars:<40}| {value:8.1f}")
+    return "\n".join(lines)
+
+
+def render(row: dict) -> str:
+    r, g, b = int(row["r"]), int(row["g"]), int(row["b"])
+    channels = {band: row.get(band) for band in BANDS}
+    return "\n".join(
+        [
+            f"  Aeg:     {row['ts']}",
+            f"  Värvus:  {row['rgb_hex']}  x={row['x_chrom']:.3f} y={row['y_chrom']:.3f}",
+            "",
+            swatch(r, g, b),
+            "",
+            "  Spekter (kanalid):",
+            spectrum_bar(channels),
+        ]
+    )
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="AS7341 reaalajas värvinäitaja")
+    parser.add_argument("--interval", type=float, default=2.0, help="värskendus sekundites")
+    parser.add_argument("--once", action="store_true", help="kuva üks kord ja välju")
+    args = parser.parse_args()
+    try:
+        while True:
+            row = fetch_latest()
+            os.system("clear")
+            print("AS7341 värvinäitaja  (Ctrl+C väljumiseks)")
+            print("=" * 54)
+            print(render(row) if row else "  Andmeid pole.")
+            if args.once:
+                break
+            time.sleep(args.interval)
+    except KeyboardInterrupt:
+        pass
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

dev/varvusensor/collector/requirements.txt

@@ -0,0 +1,2 @@
+paho-mqtt>=2.0,<3
+requests>=2.31

dev/varvusensor/collector/settings.py

@@ -0,0 +1,45 @@
+"""Ühised seaded CH-tööriistadele (live.py, calibrate.py jne).
+
+Loeb .env failist (kui olemas) kasutaja/parooli, siis keskkonnamuutujatest.
+CH_URL jäetakse .env-ist vahele (seal on see konteineri jaoks `clickhouse`);
+vaikimisi kasutame Tailscale IP-d, mis töötab nii ohmu hostil kui mujal
+tailnetis. Vajadusel saab CH_URL-i ikka keskkonnamuutujaga üle kirjutada.
+
+NB! Ilma auth-ita ei saa: sea CH_USER/CH_PASS (või pane need .env-i).
+"""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+_ENV_CANDIDATES = (
+    Path(__file__).resolve().parents[1] / ".env",        # varvusensor/.env
+    Path("/opt/docker-compose/dev/varvusensor/.env"),     # ohmu live-stack
+)
+# Konteinerispetsiifilised võtmed, mida host-tööriistadele ei laadita.
+_SKIP_FROM_ENV = {"CH_URL", "MQTT_HOST", "MQTT_PORT", "MQTT_TOPIC"}
+
+
+def _load_env() -> None:
+    for path in _ENV_CANDIDATES:
+        if not path.is_file():
+            continue
+        for line in path.read_text().splitlines():
+            line = line.strip()
+            if not line or line.startswith("#") or "=" not in line:
+                continue
+            key, _, value = line.partition("=")
+            key = key.strip()
+            if key in _SKIP_FROM_ENV:
+                continue
+            os.environ.setdefault(key, value.strip())
+        break
+
+
+_load_env()
+
+CH_URL = os.environ.get("CH_URL", "http://100.87.1.24:8125").rstrip("/")
+CH_DB = os.environ.get("CH_DB", "uus67")
+CH_TABLE = os.environ.get("CH_TABLE", "varvus")
+CH_USER = os.environ.get("CH_USER", "zigbee")
+CH_PASS = os.environ.get("CH_PASS", "")

dev/varvusensor/config/.gitignore

@@ -0,0 +1,5 @@
+# Gitignore settings for ESPHome
+# This is an example and may include too much for your use-case.
+# You can modify this file to suit your needs.
+/.esphome/
+/secrets.yaml

dev/varvusensor/config/varvusensor.yaml

@@ -0,0 +1,212 @@
+substitutions:
+  name: varvusensor
+  friendly_name: Varvusensor
+  mqtt_prefix: varvus
+  gain_num: 256
+
+esphome:
+  name: ${name}
+  friendly_name: ${friendly_name}
+
+esp32:
+  board: esp32-c6-devkitc-1
+  variant: esp32c6
+  framework:
+    type: esp-idf
+
+logger:
+  level: DEBUG
+  hardware_uart: USB_SERIAL_JTAG
+
+api:
+  encryption:
+    key: !secret api_key
+
+ota:
+  - platform: esphome
+    password: !secret fallback_ap_password
+
+wifi:
+  networks:
+    - ssid: !secret wifi_ssid
+      password: !secret wifi_password
+    - ssid: "SIPELGAS"
+      password: "misonminuwifinimi"
+    - ssid: "Ristumine peateega"
+      password: "Mikihiir"
+  ap:
+    ssid: "${friendly_name} Fallback"
+    password: !secret fallback_ap_password
+
+captive_portal:
+
+# Avalik MQTT broker edge-serveril (TLS 8883) — toimib igast võrgust.
+mqtt:
+  broker: mqtt.odamus.com
+  port: 8883
+  username: !secret mqtt_user
+  password: !secret mqtt_pass
+  certificate_authority: |-
+    -----BEGIN CERTIFICATE-----
+    MIICizCCAhGgAwIBAgIQXd1w3TH4AchcGGp6BLgK/jAKBggqhkjOPQQDAzAuMQsw
+    CQYDVQQGEwJVUzENMAsGA1UEChMESVNSRzEQMA4GA1UEAxMHUm9vdCBZRTAeFw0y
+    NTA5MDMwMDAwMDBaFw0yODA5MDIyMzU5NTlaMDMxCzAJBgNVBAYTAlVTMRYwFAYD
+    VQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQDEwNZRTEwdjAQBgcqhkjOPQIBBgUr
+    gQQAIgNiAAQHZVB1/mimla2hfSurylScjPMZaOJXLz/NnAc2sylm8WDyhU9Ccp+z
+    ASQi5vSwGGJjSGklkD9fdPR8GpyDIOIjCEfrnbt/v+ZSEPLLEGbaM6EccDbN7p9x
+    teIm2Avf+ryjge4wgeswDgYDVR0PAQH/BAQDAgGGMBMGA1UdJQQMMAoGCCsGAQUF
+    BwMBMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFLsgykcL/tflnPmPCSqj
+    jDdFsbzYMB8GA1UdIwQYMBaAFKPIJlqOoUzQNWP8myPIOq5W809WMDIGCCsGAQUF
+    BwEBBCYwJDAiBggrBgEFBQcwAoYWaHR0cDovL3llLmkubGVuY3Iub3JnLzATBgNV
+    HSAEDDAKMAgGBmeBDAECATAnBgNVHR8EIDAeMBygGqAYhhZodHRwOi8veWUuYy5s
+    ZW5jci5vcmcvMAoGCCqGSM49BAMDA2gAMGUCMQDgjUEahFT/h3DRakqiPZpLvPgf
+    Zwkt6K2EOMmh1nvEzl83eMLYcod4GCl3b0J1Nn0CMBNYmEQJb4CEG5WoOe7aRn/L
+    VKu6saHmHEynI7ysIPd8zQsK1HdmhlHKlw9Z5GpGvA==
+    -----END CERTIFICATE-----
+    -----BEGIN CERTIFICATE-----
+    MIICpjCCAiugAwIBAgIRAIchZfw0tuX7qK3Vs3BftTowCgYIKoZIzj0EAwMwTzEL
+    MAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2VhcmNo
+    IEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDIwHhcNMjYwNTEzMDAwMDAwWhcN
+    MzIwOTAyMjM1OTU5WjAuMQswCQYDVQQGEwJVUzENMAsGA1UEChMESVNSRzEQMA4G
+    A1UEAxMHUm9vdCBZRTB2MBAGByqGSM49AgEGBSuBBAAiA2IABDwS/6vhrcVqcbBo
+    +wgdI3fwn9x7DNJJOY/lTOti0vkwuRN87RhEhTH17E7XyFjWsPYhIPt/wzOqxTd2
+    b+4ZJNy9ID04YywF9U5zasDVyGSNErVNtz8uSGh5izW87j77GaOB6zCB6DAOBgNV
+    HQ8BAf8EBAMCAQYwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUwAwEB
+    /zAdBgNVHQ4EFgQUo8gmWo6hTNA1Y/ybI8g6rlbzT1YwHwYDVR0jBBgwFoAUfEKW
+    rt5LSDv6kviejM9ti6lyN5UwMgYIKwYBBQUHAQEEJjAkMCIGCCsGAQUFBzAChhZo
+    dHRwOi8veDIuaS5sZW5jci5vcmcvMBMGA1UdIAQMMAowCAYGZ4EMAQIBMCcGA1Ud
+    HwQgMB4wHKAaoBiGFmh0dHA6Ly94Mi5jLmxlbmNyLm9yZy8wCgYIKoZIzj0EAwMD
+    aQAwZgIxAMU19WCtmxVND8UHBZRoma49Z7jPs64Dma0eTu1OChVbB/2J7GV3nvYK
+    Ax54uk1G9QIxAO0miLVJu8PLNiXXXkiE/gsK3CTRTF/aeo4bMX42Zw40csRU6AC2
+    6hSW1/IWaas6dg==
+    -----END CERTIFICATE-----
+    -----BEGIN CERTIFICATE-----
+    MIIEcDCCAligAwIBAgIQbI8dxyfHEX97r4U6yYD5zTANBgkqhkiG9w0BAQsFADBP
+    MQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJuZXQgU2VjdXJpdHkgUmVzZWFy
+    Y2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBYMTAeFw0yNjA1MTMwMDAwMDBa
+    Fw0zMjA5MDIyMzU5NTlaME8xCzAJBgNVBAYTAlVTMSkwJwYDVQQKEyBJbnRlcm5l
+    dCBTZWN1cml0eSBSZXNlYXJjaCBHcm91cDEVMBMGA1UEAxMMSVNSRyBSb290IFgy
+    MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEzZvVn4CDCuwJSvMWSj5cz3es3mcFDR0H
+    ttwW+1qLFNvicWDEukWVEYmO6gbf9yoWHKS5xcUy4APgHoIYOIvXRdgKam7mAHf7
+    AlF9ItgKbppbd9/w+kHsOdx1ymgHDB/qo4H1MIHyMA4GA1UdDwEB/wQEAwIBBjAd
+    BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUwAwEB/zAd
+    BgNVHQ4EFgQUfEKWrt5LSDv6kviejM9ti6lyN5UwHwYDVR0jBBgwFoAUebRZ5nu2
+    5eQBc4AIiMgaWPbpm24wMgYIKwYBBQUHAQEEJjAkMCIGCCsGAQUFBzAChhZodHRw
+    Oi8veDEuaS5sZW5jci5vcmcvMBMGA1UdIAQMMAowCAYGZ4EMAQIBMCcGA1UdHwQg
+    MB4wHKAaoBiGFmh0dHA6Ly94MS5jLmxlbmNyLm9yZy8wDQYJKoZIhvcNAQELBQAD
+    ggIBAD2/e9frmMxNpCV03qUHegg+MV2wz9644YoXdqtH8RyWYcBO7xfjjGEXdU1e
+    /o0OkEFiynUCOSIk/vLLo7ttz6CPAeNlWfC0XNkoGeWgK6jjXvozBaGuGH5n0Ufo
+    shMeWTuURqNN5G00sSXDTBrpp2+mgvdZQjb8K11TYMA25QA+YHNfbIEL0BniAhKS
+    2gsnJjSzrdZLI+EZ7SEyqdR2rkjd1KutLDU+n3TFyxjniZVGur4YlhMP3mY/dV95
+    IruAkkjOZier6hGBdEgZXXvaCz9u9iVEadsIE75pAGL8oHV5vxdARDiotRpul1IN
+    /UZwzAbrfUFcw1HkAcYD/mlZfnQ2ieCF2MS7j3Vhv7JPDKp45fmykmzYNSrumRW0
+    upFFKDBOoF7hsOb7oLyHS+Uft6jOUfOrogj8YUx38hKb2K20r42OgsSdDdxdeYWc
+    MS3Sb6mwJeSZEYxJ2gaXnDSPaKhhrNkYwljyVQyr4Nq+MEJytXNTnHqaAcrNwZlV
+    pcJL1KBnMrMjP7eanvUwL3FYj3cF17jtboLt7gLoi4+2rWZFvn+w54jmd/FIuhhZ
+    cEaU/wvU6BUNMtcVquVGHp7itQeDth5j+XL3j4WJ2SABwzUl6OeYdgpIt/ITZa+p
+    TT0mQ/r5XyA4MEAiabn7XJjvCERlF2dcn2wqJw+CreTkkQ2R
+    -----END CERTIFICATE-----
+    -----BEGIN CERTIFICATE-----
+    MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
+    TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
+    cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
+    WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
+    ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
+    MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
+    h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
+    0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
+    A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
+    T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
+    B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
+    B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
+    KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
+    OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
+    jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
+    qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
+    rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
+    HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
+    hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
+    ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
+    3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
+    NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
+    ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
+    TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
+    jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
+    oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
+    4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
+    mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
+    emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
+    -----END CERTIFICATE-----
+
+i2c:
+  sda: GPIO6
+  scl: GPIO7
+  scan: true
+  frequency: 100kHz
+
+sensor:
+  # AS7341 saadab madala baidi esimesena (little-endian), aga ESPHome i2ctohs
+  # vahetab baidid -> lisame lambdas vahetuse tagasi, et saada toelised 16-bit vaartused.
+  - platform: as7341
+    gain: X256
+    f1:
+      name: "${friendly_name} F1 415nm"
+      id: ch_f1
+      filters: &as7341_swap
+        - lambda: |-
+            uint16_t v = (uint16_t) x;
+            return (float) (uint16_t) ((v >> 8) | ((v & 0xFF) << 8));
+    f2:
+      name: "${friendly_name} F2 445nm"
+      id: ch_f2
+      filters: *as7341_swap
+    f3:
+      name: "${friendly_name} F3 480nm"
+      id: ch_f3
+      filters: *as7341_swap
+    f4:
+      name: "${friendly_name} F4 515nm"
+      id: ch_f4
+      filters: *as7341_swap
+    f5:
+      name: "${friendly_name} F5 555nm"
+      id: ch_f5
+      filters: *as7341_swap
+    f6:
+      name: "${friendly_name} F6 590nm"
+      id: ch_f6
+      filters: *as7341_swap
+    f7:
+      name: "${friendly_name} F7 630nm"
+      id: ch_f7
+      filters: *as7341_swap
+    f8:
+      name: "${friendly_name} F8 680nm"
+      id: ch_f8
+      filters: *as7341_swap
+    clear:
+      name: "${friendly_name} Clear"
+      id: ch_clear
+      filters: *as7341_swap
+    nir:
+      name: "${friendly_name} NIR"
+      id: ch_nir
+      filters: *as7341_swap
+
+interval:
+  - interval: 5s
+    then:
+      - mqtt.publish_json:
+          topic: ${mqtt_prefix}/spectral
+          payload: |-
+            if (!isnan(id(ch_f1).state))    root["f1"]    = id(ch_f1).state;
+            if (!isnan(id(ch_f2).state))    root["f2"]    = id(ch_f2).state;
+            if (!isnan(id(ch_f3).state))    root["f3"]    = id(ch_f3).state;
+            if (!isnan(id(ch_f4).state))    root["f4"]    = id(ch_f4).state;
+            if (!isnan(id(ch_f5).state))    root["f5"]    = id(ch_f5).state;
+            if (!isnan(id(ch_f6).state))    root["f6"]    = id(ch_f6).state;
+            if (!isnan(id(ch_f7).state))    root["f7"]    = id(ch_f7).state;
+            if (!isnan(id(ch_f8).state))    root["f8"]    = id(ch_f8).state;
+            if (!isnan(id(ch_clear).state)) root["clear"] = id(ch_clear).state;
+            if (!isnan(id(ch_nir).state))   root["nir"]   = id(ch_nir).state;
+            root["gain"] = ${gain_num};

dev/varvusensor/docker-compose.yml

@@ -0,0 +1,40 @@
+services:
+  # Kohalik MQTT-broker (ESP32 jaoks, kuna valvuri broker on Tailscale'i taga).
+  mosquitto:
+    image: eclipse-mosquitto:2
+    container_name: varvus-mosquitto
+    restart: unless-stopped
+    mem_limit: 64m
+    user: "1883:1883"
+    environment:
+      - TZ=Europe/Tallinn
+    ports:
+      - "192.168.68.114:1883:1883"
+      - "100.87.1.24:1883:1883"
+    volumes:
+      - ./mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro
+      - ./mosquitto/acl:/mosquitto/config/acl:ro
+      - ./mosquitto/passwd:/mosquitto/config/passwd:ro
+      - ./data/mosquitto:/mosquitto/data
+    networks:
+      - main_ai_network
+
+  # AS7341 värvianduri koguja: MQTT -> ClickHouse (uus67.varvus).
+  varvus-collector:
+    build: ./collector
+    container_name: varvus-collector
+    restart: unless-stopped
+    mem_limit: 128m
+    env_file:
+      - .env
+    environment:
+      - TZ=Europe/Tallinn
+    depends_on:
+      - mosquitto
+    networks:
+      - main_ai_network
+
+networks:
+  main_ai_network:
+    external: true
+    name: main_ai_network

dev/varvusensor/edge-mqtt/acl

@@ -0,0 +1,3 @@
+# mqtt-mosquitto ACL — mqtt_user pääseb ainult varvus-teemadele
+user mqtt_user
+topic readwrite varvus/#

dev/varvusensor/edge-mqtt/docker-compose.yml

@@ -0,0 +1,19 @@
+services:
+  # Avalik MQTT broker (TLS 8883) varvusensorile ja teistele.
+  # Sert pärineb Caddyst (edge_caddy), sünkroonitakse sync-certs.sh-iga.
+  mosquitto:
+    image: eclipse-mosquitto:2
+    container_name: mqtt-mosquitto
+    restart: unless-stopped
+    mem_limit: 64m
+    user: "1883:1883"
+    environment:
+      - TZ=Europe/Tallinn
+    ports:
+      - "8883:8883"
+    volumes:
+      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf:ro
+      - ./passwd:/mosquitto/config/passwd:ro
+      - ./acl:/mosquitto/config/acl:ro
+      - ./certs:/mosquitto/certs:ro
+      - ./data:/mosquitto/data

dev/varvusensor/edge-mqtt/mosquitto.conf

@@ -0,0 +1,12 @@
+listener 8883 0.0.0.0
+allow_anonymous false
+password_file /mosquitto/config/passwd
+acl_file /mosquitto/config/acl
+certfile /mosquitto/certs/server.crt
+keyfile /mosquitto/certs/server.key
+persistence true
+persistence_location /mosquitto/data/
+log_dest stdout
+log_type error
+log_type warning
+log_type notice

dev/varvusensor/edge-mqtt/sync-certs.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+# Sünkroonib Caddy (edge_caddy) Let's Encrypt serdi mqtt.odamus.com jaoks
+# mosquitto jaoks loetavaks. Käivitub cronist (iga päev). Kui sert muutus,
+# taaskäivitab mosquitto, et uus sert laaditaks.
+#
+# NB: LE ahel lõpeb ISRG Root X2-ga, mis on rist-signeeritud ISRG Root X1-ga
+# (self-signed X1 ei ole ahelas). Seetõttu lisame CA-hunnikusse ka ISRG Root X1.
+set -euo pipefail
+
+SRC=/var/lib/docker/volumes/edge_caddy_data/_data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/mqtt.odamus.com
+BASE=/opt/mqtt
+
+mkdir -p "$BASE/certs"
+
+NEW_HASH=$(sha256sum "$SRC/mqtt.odamus.com.crt" | cut -d' ' -f1)
+OLD_HASH=$(cat "$BASE/certs/.cert.hash" 2>/dev/null || echo none)
+
+cp "$SRC/mqtt.odamus.com.crt" "$BASE/certs/server.crt"
+cp "$SRC/mqtt.odamus.com.key" "$BASE/certs/server.key"
+
+# ISRG Root X1 (self-signed, kehtiv kuni 2035) — vajalik ahela lõpetamiseks.
+if [ ! -s "$BASE/certs/isrgrootx1.pem" ]; then
+  curl -fsS https://letsencrypt.org/certs/isrgrootx1.pem -o "$BASE/certs/isrgrootx1.pem" || true
+fi
+
+python3 - <<'PY'
+import re
+
+data = open("/opt/mqtt/certs/server.crt").read()
+certs = re.findall(r"-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----", data, re.S)
+chain = "\n".join(certs[1:]) + "\n"
+try:
+    root = open("/opt/mqtt/certs/isrgrootx1.pem").read()
+except FileNotFoundError:
+    root = ""
+open("/opt/mqtt/certs/esp32-ca.pem", "w").write(chain + root)
+print(f"ahelas {len(certs)} serti + ISRG Root X1")
+PY
+
+echo "$NEW_HASH" > "$BASE/certs/.cert.hash"
+chown -R 1883:1883 "$BASE/certs"
+chmod 644 "$BASE/certs/server.crt" "$BASE/certs/esp32-ca.pem"
+chmod 600 "$BASE/certs/server.key"
+
+if [ "$NEW_HASH" != "$OLD_HASH" ]; then
+  docker restart mqtt-mosquitto >/dev/null 2>&1 || true
+fi
+echo "sert sünkroonitud; hash=$NEW_HASH"

dev/varvusensor/mosquitto/acl

@@ -0,0 +1,3 @@
+# varvus-mosquitto ACL — mqtt_user pääseb ainult varvus-teemadele
+user mqtt_user
+topic readwrite varvus/#

dev/varvusensor/mosquitto/mosquitto.conf

@@ -0,0 +1,10 @@
+listener 1883 0.0.0.0
+allow_anonymous false
+password_file /mosquitto/config/passwd
+persistence true
+persistence_location /mosquitto/data/
+log_dest stdout
+log_type error
+log_type warning
+log_type notice
+acl_file /mosquitto/config/acl