Algus

Dockerfile/rag-api/.dockerignore

@@ -0,0 +1,4 @@
+# välista PyTorch workspace ja prügikast
+main/pytorch-workspace
+**/.Trash-0
+**/.Trash-*

Dockerfile/rag-api/Dockerfile

@@ -0,0 +1,15 @@
+FROM python:3.12-slim
+
+# Build-tools
+RUN apt-get update && apt-get install -y \
+    build-essential g++ gcc \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+RUN pip install --upgrade pip
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# MUUDA SEDA RIDA:
+CMD ["python", "-m", "uvicorn", "src.rag_api:app", "--host", "0.0.0.0", "--port", "8000"]

Dockerfile/rag-api/LOEMIND.md

@@ -0,0 +1,19 @@
+
+cd /opt/docker-compose/rag-api
+
+cat > .dockerignore << 'EOF'
+# välista PyTorch workspace ja prügikast
+main/pytorch-workspace
+**/.Trash-0
+**/.Trash-*
+EOF
+
+
+docker build -t rag-api .
+docker run -p 8072:8000 rag-api
+
+docker run --rm -p 8072:8000 \
+  -v ~/rag-demo/pdf-pipeline/src:/app/src \
+  rag-api \
+  uvicorn src.rag_api:app --host 0.0.0.0 --port 8000
+

Dockerfile/rag-api/requirements.txt

@@ -0,0 +1,14 @@
+fastapi==0.109.0
+uvicorn==0.27.0
+pydantic==2.12.0
+python-multipart==0.0.6
+langchain==0.1.7
+ollama==0.6.0
+openai==1.3.9
+weaviate-client==4.19.0
+pypdf==4.0.1
+python-docx==1.0.1
+requests==2.31.0
+python-dotenv==1.0.0
+psycopg2-binary
+sentence_transformers

LOEMIND.md

@@ -0,0 +1,48 @@
+## Kasutamine
+
+```bash
+# Käivita Portainer
+docker-compose up -d
+# Vaata logisid
+docker-compose logs -f portainer
+# Peata
+docker-compose down
+# Eemalda andmed
+#docker-compose down -v 2>/dev/null || true
+```
+
+## Kiire test
+```bash
+# PostgreSQL test
+docker exec -it postgres_postgis psql -U ai_user -d research_db -c "SELECT PostGIS_Version();"
+# Weaviate test
+curl http://localhost:8080/v1/.well-known/ready
+# Jupyter token
+docker logs jupyter | grep token=
+# Ollama test
+curl http://localhost:11434/api/tags
+# Vaata volume's
+docker volume ls
+```
+
+## 1 teenuse käsud
+```bash
+# Uuenda ja taaskäivita Portainer
+docker-compose up -d portainer
+# Vaata logisid
+docker-compose logs -f portainer
+# Kontrolli staatus pärast ~60 sekundit
+docker-compose ps
+```
+
+## Postgresql installimised
+./scripts/install-pgvector.sh
+
+## Build'i loomine rag-api jaoks
+docker-compose build rag-api
+docker-compose up -d rag-api
+### 3. Kontrolli logisid
+docker-compose logs -f rag-api
+### 4. Testi
+curl http://localhost:8072/health
+curl "http://localhost:8072/search-simple?q=young+driver+risk"

README/Erinevused PyTorch (ROCm) ja JupyterLab konteineritel.md

@@ -0,0 +1,165 @@
+# Erinevused PyTorch (ROCm) ja JupyterLab konteinerite vahel
+
+Suurepärane küsimus! Vaatame, mida kumbki pakub:
+
+## 🔴 **ROCm PyTorch konteiner** (`rocm_pytorch`, port 8889)
+
+### ✅ **Eelised:**
+
+- **GPU tugi AMD ROCm-iga** - kasutab sinu AMD Ryzen AI Max+ 395 GPU-d
+- **PyTorch eelinstallitud** - ROCm-iga kompileeritud, optimeeritud AMD GPU-le
+- **ML/DL raamistik** - transformers, BERT, GPT, neural networks
+- **Kiirus** - GPU kiirendus deep learning mudelitele
+- **96GB RAM juurdepääs** - kogu süsteemi RAM on kasutatav GPU tööks
+
+
+### ⚠️ **Puudused:**
+
+- **Suurem image** (~12GB) - võtab rohkem ruumi
+- **Keerulisem setup** - vajab `/dev/kfd`, `/dev/dri` device'e
+- **AMD-spetsiifiline** - ROCm töötab ainult AMD GPU-del
+- **Eksperimentaalne** - mõned funktsioonid (Flash Attention) on veel betas
+
+
+### 🎯 **Kasuta, kui:**
+
+- Teed **deep learning** (neural networks, transformers, LLM-id)
+- Vajad **GPU kiirendust** (suur andmehulk, suured mudelid)
+- Töötad **embeddings/vektorite** genereerimisega
+- Treenid **mudeleid** (fine-tuning, transfer learning)
+- Teadusartiklite **NLP analüüs** (BERT, SciBERT, BioBERT)
+
+***
+
+## 🔵 **JupyterLab konteiner** (`jupyter`, port 8888)
+
+### ✅ **Eelised:**
+
+- **Kerge** (~2GB image) - väiksem, kiirem käivitamine
+- **Laialdane toetus** - pandas, numpy, scipy, matplotlib, scikit-learn
+- **Andmeanalüüs** - statistika, visualiseerimine, exploratory data analysis
+- **Lihtne** - ei vaja GPU seadistust
+- **Stabiilne** - küps ökosüsteem
+
+
+### ⚠️ **Puudused:**
+
+- **Ei toeta GPU-d** - töötab ainult CPU-l
+- **Aeglasem deep learning** - ilma GPU-ta 10-100x aeglasem
+- **Ei ole PyTorch** - pead installeerima (aga CPU versioon)
+- **Piiratud ML** - scikit-learn on olemas, aga deep learning puudub
+
+
+### 🎯 **Kasuta, kui:**
+
+- Teed **andmeanalüüsi** (pandas, numpy, statistika)
+- **Visualiseerimine** (matplotlib, seaborn, plotly)
+- **Klassikaline ML** (scikit-learn: regression, clustering, classification)
+- **Kerge töö** - tabeliandmed, CSV, SQL päringud
+- **Ei vaja GPU-d** - väikesed andmehulgad
+
+***
+
+## 📊 Võrdlustabel
+
+| Kriteerium | ROCm PyTorch (port 8889) | JupyterLab (port 8888) |
+| :-- | :-- | :-- |
+| **GPU tugi** | ✅ AMD ROCm | ❌ Ainult CPU |
+| **PyTorch** | ✅ Eelinstallitud | ❌ Puudub |
+| **Deep Learning** | ✅ Neural networks | ❌ Ainult scikit-learn |
+| **Transformers** | ✅ BERT, GPT, LLM-id | ❌ Puudub |
+| **Pandas/NumPy** | ✅ Installitav | ✅ Eelinstallitud |
+| **Kiirus (DL)** | 🚀 GPU kiirendus | 🐌 CPU (10-100x aeglasem) |
+| **Image suurus** | ~12GB | ~2GB |
+| **Kasutusvaldkond** | ML/DL, NLP, Computer Vision | Andmeanalüüs, statistika |
+| **Stabiilsus** | ⚠️ Eksperimentaalne | ✅ Väga stabiilne |
+
+
+***
+
+## 🎯 Praktiline soovitus sinu projektile:
+
+### **Kasuta ROCm PyTorch (port 8889), kui:**
+
+```python
+# Teadusartiklite embeddingi-d
+from transformers import AutoModel
+model = AutoModel.from_pretrained("allenai/scibert_scivocab_uncased").to("cuda")
+
+# Neural network treenimine
+import torch.nn as nn
+model = MyNeuralNetwork().to("cuda")
+
+# Suur andmehulk
+for batch in large_dataset:
+    embeddings = model(batch.to("cuda"))
+```
+
+
+### **Kasuta JupyterLab (port 8888), kui:**
+
+```python
+# Andmeanalüüs
+import pandas as pd
+df = pd.read_csv('articles.csv')
+df.groupby('year').size().plot()
+
+# Statistika
+from scipy import stats
+stats.ttest_ind(group1, group2)
+
+# Klassikaline ML
+from sklearn.ensemble import RandomForestClassifier
+clf = RandomForestClassifier()
+clf.fit(X_train, y_train)
+```
+
+
+***
+
+## 💡 **Sinu projektis (teadusartiklite analüüs):**
+
+### **Ideaalne töövoog:**
+
+1. **ROCm PyTorch** → Genereeri embeddingi-d GPU-ga (kiire!)
+
+```python
+# port 8889
+embeddings = bert_model(articles).to("cuda")
+save_to_postgres(embeddings)
+```
+
+2. **JupyterLab** → Analüüsi tulemusi (pandas, plotly)
+
+```python
+# port 8888
+df = pd.read_sql("SELECT * FROM articles", conn)
+df['year'].value_counts().plot(kind='bar')
+```
+
+
+***
+
+## Kokkuvõte:
+
+| Stsenaarium | Konteiner |
+| :-- | :-- |
+| 🧠 **Deep learning, transformers, embeddingi-d** | ROCm PyTorch ✅ |
+| 📊 **Andmeanalüüs, visualiseerimine, statistika** | JupyterLab ✅ |
+| 🚀 **Suur andmehulk, mudelite treenimine** | ROCm PyTorch ✅ |
+| 📈 **SQL päringud, tabelid, graafikud** | JupyterLab ✅ |
+
+**Sinu projektis saad kasutada mõlemat paralleelselt:**
+
+- **ROCm PyTorch** → GPU-intensiivne töö (embeddingi-d)
+- **JupyterLab** → Kerge analüüs ja visualiseerimine
+
+**Kas on veel küsimusi Docker setup'i kohta?** 🚀
+<span style="display:none">[^1][^2]</span>
+
+<div align="center">⁂</div>
+
+[^1]: image.jpg
+
+[^2]: image.jpg
+

README/README_RAG-API.md

@@ -0,0 +1,118 @@
+<img src="https://r2cdn.perplexity.ai/pplx-full-logo-primary-dark%402x.png" style="height:64px;margin-right:32px"/>
+
+# Super. Aitäh. Kõik töötab!!!!
+
+🎉🎉🎉 **SUUREPÄRANE!** Palju õnne!
+
+Näen, et RAG API **genereeris täielikult eesti keelses vastuse** – artikelite viidete, praktiliste rakenduste ja teaduslike järelduste abil!
+
+## ✅ Mis nüüd toimib
+
+- ✅ **Weaviate** – artikleid otsitakse ja leitakse
+- ✅ **Embeddings** – Weaviate teeb vector search'i
+- ✅ **PostgreSQL + pgvector** – pole kasutatud, aga baas on olemas
+- ✅ **llama.cpp LLM** – genereerib kvaliteetset eesti keelset teksti
+- ✅ **RAG pipeline** – konteksti ehitamine + LLM response generation
+- ✅ **Docker Compose** – kõik teenused käivad isoleeritud keskkonnaga
+
+***
+
+## 🚀 Järgmised sammud (soovitused)
+
+### 1. **Optimiseerimine**
+
+```bash
+# Kontrolli Docker ressursside kasutamist
+docker-compose stats
+```
+
+
+### 2. **Logging ja Monitoring**
+
+```bash
+# Pidevalt jälgi logisid
+docker-compose logs -f rag-api --tail 50
+
+# Või salvestada failesse
+docker-compose logs rag-api > /tmp/rag-api.log
+```
+
+
+### 3. **API dokumentatsioon**
+
+```bash
+# OpenAPI spec on olemas
+curl http://localhost:8072/openapi.json > /tmp/rag-api-openapi.json
+```
+
+
+### 4. **Backup strateegia**
+
+```bash
+# Weaviate andmete backup
+docker-compose exec weaviate tar czf - /var/lib/weaviate > /backup/weaviate-$(date +%Y%m%d).tar.gz
+
+# PostgreSQL dump
+docker-compose exec postgres pg_dump -U osm pdf_research > /backup/db-$(date +%Y%m%d).sql
+```
+
+
+### 5. **Health Check'ide parandamine**
+
+```bash
+# Kontrollida kõigi teenuste tervist
+curl http://localhost:8072/health
+curl http://localhost:8080/v1/.well-known/ready
+curl http://localhost:5432/  # PostgreSQL
+```
+
+
+***
+
+## 📋 Dokumentatsioon kokkuvõttes
+
+**Teie RAG API stack:**
+
+- 🔵 **Weaviate** (Vector DB): http://localhost:8080
+- 🟡 **llama.cpp** (LLM): http://localhost:8070
+- 🟢 **RAG API** (FastAPI): http://localhost:8072/docs
+- 🔴 **PostgreSQL** (pgvector): localhost:5432
+- 🟣 **Ollama**: http://localhost:11434 (olemas, aga ei kasuta)
+
+***
+
+**Palju edu edaspidises tööga!** 🚀 Kui sul on küsimusi või soovid midagi parandada, olen siin! 💪
+<span style="display:none">[^1][^10][^11][^12][^13][^14][^15][^2][^3][^4][^5][^6][^7][^8][^9]</span>
+
+<div align="center">⁂</div>
+
+[^1]: response_1769191939220.json
+
+[^2]: requirements.txt
+
+[^3]: image.jpg
+
+[^4]: weaviate_client.py
+
+[^5]: image.jpg
+
+[^6]: image.jpg
+
+[^7]: docker-compose.yml
+
+[^8]: image.jpg
+
+[^9]: image.jpg
+
+[^10]: docker-compose.yml
+
+[^11]: rag_api.py
+
+[^12]: query_hybrid.py
+
+[^13]: docker-compose.yml
+
+[^14]: response_1769185866090.json
+
+[^15]: generate_answer.py
+

cleanup.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+echo "Cleaning up..."
+
+docker-compose down
+
+exit 0

docker-compose.yml

@@ -0,0 +1,197 @@
+version: '3.8'
+
+services:
+  portainer:
+    image: portainer/portainer-ce:latest
+    container_name: portainer
+    restart: always
+    ports:
+      - "8000:8000"
+      - "9000:9000"
+      - "9443:9443"
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - portainer_data:/data
+    environment:
+      - TZ=Europe/Tallinn
+    networks:
+      - ai_network
+
+  postgres:
+    image: kartoza/postgis:16
+    container_name: postgres_postgis
+    restart: always
+    ports:
+      - "5432:5432"
+    environment:
+      POSTGRES_USER: ai_user
+      POSTGRES_PASSWORD: ai_secure_pass_2026
+      POSTGRES_DB: research_db
+      POSTGRES_INITDB_ARGS: "-c shared_buffers=256MB -c max_connections=200"
+      PGDATA: /var/lib/postgresql/data/pgdata
+      TZ: Europe/Tallinn
+    volumes:
+      - postgres_data:/var/lib/postgresql/data
+      - ./init-db:/docker-entrypoint-initdb.d
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ai_user -d research_db"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    networks:
+      - ai_network
+
+  weaviate:
+    image: semitechnologies/weaviate:latest
+    container_name: weaviate
+    restart: always
+    ports:
+      - "8080:8080"
+      - "50051:50051"
+    environment:
+      QUERY_DEFAULTS_LIMIT: 25
+      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
+      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
+      DEFAULT_VECTORIZER_MODULE: 'none'
+      CLUSTER_HOSTNAME: 'weaviate'
+    volumes:
+      - weaviate_data:/var/lib/weaviate
+    networks:
+      - ai_network
+
+  ollama:
+    image: ollama/ollama:latest
+    container_name: ollama
+    restart: always
+    ports:
+      - "11434:11434"
+    volumes:
+      - ollama_data:/root/.ollama
+    environment:
+      - OLLAMA_HOST=0.0.0.0:11434
+      - NVIDIA_VISIBLE_DEVICES=all
+      - CUDA_VISIBLE_DEVICES=0
+    devices:
+      - /dev/kfd:/dev/kfd
+      - /dev/dri:/dev/dri
+    networks:
+      - ai_network
+
+  jupyter:
+    image: jupyter/datascience-notebook:latest
+    container_name: jupyter
+    restart: always
+    ports:
+      - "8888:8888"
+    environment:
+      - JUPYTER_ENABLE_LAB=yes
+      - JUPYTER_TOKEN=ai_server_2026
+      - TZ=Europe/Tallinn
+    volumes:
+      - ./notebooks:/home/jovyan/work
+      - ./data:/home/jovyan/data
+      - jupyter_data:/home/jovyan/.jupyter
+    networks:
+      - ai_network
+    command: start-notebook.sh --NotebookApp.token='ai_server_2026' --NotebookApp.allow_root=True
+
+  # ROCm PyTorch - GPU-kiirendatud ML/DL
+  rocm-pytorch:
+    image: rocm/pytorch:latest
+    container_name: rocm_pytorch
+    restart: always
+    ports:
+      - "8889:8888"
+      - "6006:6006"
+    environment:
+      - HSA_OVERRIDE_GFX_VERSION=11.5.1
+      - PYTORCH_ROCM_ARCH=gfx1151
+      - ROCM_HOME=/opt/rocm
+      - TZ=Europe/Tallinn
+      - TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1  # ← UUS!
+    devices:
+      - /dev/kfd:/dev/kfd
+      - /dev/dri:/dev/dri
+    volumes:
+      - ./pytorch-workspace:/workspace
+      - ./models:/models
+      - ./data:/data
+      - rocm_pytorch_data:/root/.cache
+    working_dir: /workspace
+    cap_add:
+      - SYS_PTRACE
+    security_opt:
+      - seccomp=unconfined
+    ipc: host
+    shm_size: 8G
+    group_add:
+      - video
+    networks:
+      - ai_network
+    command: >
+      bash -c "pip install jupyter notebook tensorboard transformers datasets huggingface_hub scikit-learn pandas psycopg2-binary &&
+               jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token='ai_pytorch_2026'"
+
+  # ===== RAG API =====
+  rag-api:
+    #image: rag-api:latest
+    build:
+        #context: /home/ardo/rag-demo/pdf-pipeline
+        context: /opt/docker-compose/rag-api
+        dockerfile: Dockerfile
+    container_name: rag_api
+    restart: always
+    ports:
+        - "8072:8000"
+    environment:
+        LLAMA_CPP_URL: http://100.87.1.24:8070/completion
+        WEAVIATE_HOST: weaviate
+        WEAVIATE_HTTP_PORT: 8080
+        WEAVIATE_URL: http://weaviate:8080
+        WEAVIATE_CLASS: ScientificArticle
+        DB_HOST: postgres
+        DB_PORT: 5432
+        DB_NAME: pdf_research
+        DB_USER: osm
+        DB_PASSWORD: osm
+        OLLAMA_URL: http://ollama:11434
+        TZ: Europe/Tallinn
+    volumes:
+        - /home/ardo/rag-demo/pdf-pipeline/src:/app/src
+        - /home/ardo/rag-demo/pdf-pipeline/data:/app/data
+        - /home/ardo/rag-demo/pdf-pipeline/output:/app/output
+    depends_on:
+        - weaviate
+        - ollama
+        - postgres
+    networks:
+        - ai_network
+    healthcheck:
+        test: ["CMD", "curl", "-f", "http://localhost:8000/docs"]
+        interval: 30s
+        timeout: 10s
+        retries: 3
+        start_period: 40s
+
+volumes:
+  portainer_data:
+    driver: local
+  
+  postgres_data:
+    driver: local
+  
+  weaviate_data:
+    driver: local
+  
+  ollama_data:
+    driver: local
+  
+  jupyter_data:
+    driver: local
+  
+  rocm_pytorch_data:
+    driver: local
+
+networks:
+  ai_network:
+    driver: bridge

init-db/01-init-postgis.sql

@@ -0,0 +1,29 @@
+-- Loo lihtsalt UUID ilma extensionita
+CREATE TABLE articles (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    title TEXT NOT NULL,
+    abstract TEXT,
+    content TEXT,
+    authors TEXT[],
+    published_date DATE,
+    doi TEXT UNIQUE,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE TABLE article_embeddings (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    article_id UUID NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
+    embedding_model TEXT,
+    vector_data BYTEA,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+
+-- Indeksid
+CREATE INDEX idx_articles_title ON articles USING GIN(to_tsvector('english', title));
+CREATE INDEX idx_articles_abstract ON articles USING GIN(to_tsvector('english', abstract));
+CREATE INDEX idx_article_embeddings_article_id ON article_embeddings(article_id);
+
+-- Õigused
+GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO ai_user;
+GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO ai_user;

run.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+docker-compose up -d
+
+# Install postgresql modules
+sleep 3
+bash scripts/install-pgvector.sh
+
+exit 0

scripts/install-pgvector.sh

@@ -0,0 +1,55 @@
+# file: install-pgvector.sh
+
+#!/bin/bash
+
+echo "🔧 Paigaldame pgvector'i PostgreSQL konteineri sees..."
+
+docker exec postgres_postgis bash -c '
+apt-get update
+apt-get install -y build-essential postgresql-server-dev-16 git
+
+echo "pgvector paigaldus lõpetatud!"
+'
+
+docker exec postgres_postgis bash -c '
+apt-get install postgresql-16-pgvector
+echo "✓ Pgvector paigaldatud konteineri sees"
+'
+
+# Admin kasutaja lisamine
+docker exec -it postgres_postgis bash -c "
+  export PGPASSWORD='ai_secure_pass_2026' &&
+  psql -U ai_user -h localhost -d research_db -c \"
+  CREATE ROLE osm WITH LOGIN SUPERUSER CREATEDB CREATEROLE PASSWORD 'osm'\"
+"
+# PDF-de andmevbaasi loomine
+docker exec -it postgres_postgis bash -c "
+  export PGPASSWORD='ai_secure_pass_2026' &&
+  psql -U ai_user -h localhost -d research_db -c '
+  CREATE DATABASE pdf_research WITH OWNER = osm;'
+"
+# Laienduste lisamine
+docker exec -it postgres_postgis bash -c "
+  export PGPASSWORD='osm' &&
+  psql -U osm -h localhost -d pdf_research -c \"
+  CREATE EXTENSION IF NOT EXISTS postgis CASCADE;
+  CREATE EXTENSION IF NOT EXISTS pointcloud CASCADE;
+  CREATE EXTENSION IF NOT EXISTS pointcloud_postgis CASCADE;
+  CREATE EXTENSION IF NOT EXISTS fuzzystrmatch CASCADE;
+  CREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder CASCADE;
+  CREATE EXTENSION IF NOT EXISTS postgis_topology CASCADE;
+  CREATE EXTENSION IF NOT EXISTS postgis_raster CASCADE;
+  CREATE EXTENSION IF NOT EXISTS pgrouting CASCADE;
+  CREATE EXTENSION IF NOT EXISTS postgis_sfcgal CASCADE;
+  CREATE EXTENSION IF NOT EXISTS hstore CASCADE;
+  CREATE EXTENSION IF NOT EXISTS plpgsql CASCADE;
+  CREATE EXTENSION IF NOT EXISTS vector CASCADE;
+  SELECT * FROM pg_extension WHERE extname = 'vector';
+  \"
+"
+docker exec -it postgres_postgis bash -c "
+  export PGPASSWORD='osm' &&
+  psql -U osm -h localhost -d pdf_research -c '\dx'
+"
+
+echo "✓ Pgvector aktiveerimisel!"