diff --git a/.gitignore b/.gitignore index c03bef2..b8f3f64 100644 --- a/.gitignore +++ b/.gitignore @@ -12,4 +12,6 @@ data/resume.pdf data/chunks.json # Private readme folder -_readme/ \ No newline at end of file +_readme/ + +k8s/secret.yaml \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..8f7b0ab --- /dev/null +++ b/README.md @@ -0,0 +1,298 @@ +# VaultMind + +## What is VaultMind +Production-grade RAG system using Dev Doshi's resume PDF as the knowledge base. Built end-to-end covering the full MLOps stack. GitHub: github.com/DevDoshi19/VaultMind + +## Current Stack +- **LLM:** gpt-4o-mini (temperature=0) +- **Embeddings:** text-embedding-3-small +- **Pipeline:** LangGraph 7-node graph +- **Vector DB:** ChromaDB (L2 distance, threshold 1.6) +- **Keyword search:** BM25Okapi + RRF merge +- **Backend:** FastAPI + uvicorn + SlowAPI (rate limiting disabled for now) +- **Frontend:** Streamlit calling backend over HTTP via httpx +- **Containerization:** Docker Compose (two containers) +- **CI/CD:** GitHub Actions → ghcr.io +- **Tracing:** LangSmith +- **Evaluation:** RAGAS (isolated, not in main stack) + +--- + +## Current File Structure +``` +VaultMind/ +├── backend/ +│ ├── app/ +│ │ ├── __init__.py +│ │ ├── config.py ← pydantic-settings + .env +│ │ ├── state.py ← RAGState TypedDict +│ │ ├── nodes.py ← all 7 pipeline nodes +│ │ ├── graph.py ← LangGraph assembly +│ │ └── ingest.py ← PDF → chunk → embed → ChromaDB + BM25 +│ ├── api/ +│ │ ├── __init__.py +│ │ ├── main.py ← FastAPI app + lifespan + CORS +│ │ ├── routes/ +│ │ │ ├── __init__.py +│ │ │ ├── health.py ← GET /health (shallow + deep probe) +│ │ │ └── query.py ← POST /api/query +│ │ ├── middleware/ +│ │ │ ├── __init__.py +│ │ │ └── rate_limit.py ← SlowAPI (disabled in main.py for now) +│ │ └── auth/ +│ │ ├── __init__.py +│ │ └── oauth.py ← placeholder for Phase 16 +│ ├── data/ +│ │ ├── resume.pdf ← gitignored +│ │ └── chunks.json ← gitignored +│ ├── chroma_db/ ← gitignored +│ ├── assets/ +│ │ └── logo.png +│ ├── cli.py ← terminal dev tool (renamed from main.py) +│ ├── Dockerfile +│ ├── .dockerignore +│ └── requirements.txt +├── frontend/ +│ ├── streamlit_app.py ← calls backend over HTTP, no LangGraph import +│ ├── assets/ +│ │ └── logo.png +│ ├── Dockerfile +│ ├── .dockerignore +│ └── requirements.txt ← only streamlit + httpx +├── evaluation/ +│ ├── __init__.py +│ ├── test_dataset.py ← 14 ground truth Q&A pairs +│ ├── ragas_eval.py ← RAGAS evaluation script +│ └── requirements.txt ← ragas==0.1.21 pinned, separate venv +├── .github/ +│ └── workflows/ +│ └── ci.yml ← 3 jobs: backend-checks, frontend-checks, build-and-push +├── docker-compose.yml +├── .env ← gitignored +└── .gitignore +``` + +--- + +## Pipeline Architecture (7 nodes) +``` +input_guardrail_node + → pattern match + LLM safety check + → if blocked: route to END (skip everything) + +classify_query_node + → is question about Dev Doshi? yes/no + → if no: route to generate (early exit) + +retrieve_node + → hybrid search: ChromaDB semantic + BM25 keyword + RRF merge + → retrieval_k=3, similarity_threshold=1.6 + +context_guard_node + → token budget check (tiktoken, max 1000 tokens) + → drops chunks that exceed budget + +generate_node + → GPT-4o-mini with tenacity retry (3 attempts, exponential backoff) + → if not relevant or empty: early exit message + +output_guardrail_node + → PII redaction (regex) + LLM safety check + +confidence_node + → self-RAG scoring (0.0 to 1.0) + → appends warning if score < 0.7 +``` + +## RAGState fields +```python +question: str +question_is_relevant: bool +retrieved_docs: list[str] +retrieval_status: str +context_token_count: int +answer: str +confidence_score: float | None +prompt_tokens: int +completion_tokens: int +total_tokens: int +estimated_cost: float +input_blocked: bool +output_flagged: bool +``` + +--- + +## API Contract +**POST /api/query** +```json +Request: { "question": "string (1-500 chars)" } +Response: { + "answer": "string", + "confidence_score": "float | null", + "input_blocked": "bool", + "output_flagged": "bool", + "total_tokens": "int", + "estimated_cost": "float", + "retrieval_status": "string", + "retrieved_chunks": "int" +} +``` + +**GET /health** → shallow probe +**GET /health?deep=true** → checks graph compiled, returns node list + +--- + +## Docker Setup +```bash +# run everything +docker compose up --build + +# backend: http://localhost:8000 +# frontend: http://localhost:8501 + +# containers talk via Docker network +# frontend calls: http://backend:8000/api/query +# BACKEND_URL = os.getenv("BACKEND_URL", "http://127.0.0.1:8000") +``` + +Volumes mounted: +- `./backend/chroma_db:/app/chroma_db` +- `./backend/data:/app/data` + +--- + +## CI/CD +Three jobs on every push/PR to main: +- **backend-checks** — import verification for pipeline + API layer +- **frontend-checks** — import verification for streamlit + httpx +- **build-and-push** — builds Docker images, smoke tests `/health`, pushes to ghcr.io on main merge + +Images live at: +- `ghcr.io/devdoshi19/vaultmind-backend:latest` +- `ghcr.io/devdoshi19/vaultmind-frontend:latest` + +--- + +## RAGAS Scores (14 test cases) +``` +faithfulness: 0.993 +answer_relevancy: 0.906 +context_precision: 0.738 +context_recall: 0.821 +overall: 0.865 + +``` + +--- + +## Known Issues / Deferred +- SlowAPI rate limiting commented out in `main.py` — middleware import fixed but disabled +- Blocked input still continues past guardrail into classifier (conditional edge issue in graph.py) +- Duplicate chunks in ChromaDB from PDF ingestion +- `auth/oauth.py` is a placeholder — Phase 16 + +--- + +## Key Technical Decisions +- **ChromaDB returns L2 distance** — lower = more relevant, threshold 1.6 +- **BM25 requires chunks.json** — saved during ingest, can't use ChromaDB vectors +- **`lifespan()` pattern** — graph compiled once at startup, stored on `app.state.graph` +- **`run_in_executor`** — LangGraph invoke is synchronous, must run in thread pool inside async FastAPI +- **RAGAS isolated** — `evaluation/requirements.txt` only, never in backend image +- **`getattr(request.app.state, "graph", None)`** — defensive access in case lifespan failed + +--- + +## What's Next — Remaining Phases + +### Phase 14 — Kubernetes +Deploy both services as Kubernetes pods using minikube locally. + +Files to write: +- `k8s/backend-deployment.yaml` — 2 replicas, resource limits, env from secret +- `k8s/frontend-deployment.yaml` — 1 replica +- `k8s/backend-service.yaml` — ClusterIP, exposes port 8000 inside cluster +- `k8s/frontend-service.yaml` — NodePort, exposes 8501 to outside +- `k8s/secret.yaml` — OPENAI_API_KEY as K8s secret (base64 encoded) + +Key concepts to understand before starting: +- Pod vs Deployment vs Service +- Why ClusterIP for backend (internal only) and NodePort for frontend (external) +- How K8s pulls from ghcr.io (imagePullSecrets) +- Liveness vs readiness probes (already have /health and /health?deep=true) + +Run with: `minikube start` then `kubectl apply -f k8s/` + +### Phase 15 — Redis Caching + Pinecone +Two upgrades, can be done separately: + +**Redis:** +- Cache `POST /api/query` responses by question hash +- Cache key: `md5(question.lower().strip())` +- TTL: 24 hours +- Changes only `query.py` — check cache before invoking graph, write to cache after +- Add `redis` service to docker-compose.yml + +**Pinecone:** +- Replace ChromaDB (local only, can't share between K8s replicas) +- Changes: `ingest.py` push to Pinecone, `nodes.py` retrieve_node queries Pinecone +- API contract unchanged + +### Phase 16 — Auth + PostgreSQL + Per-user history +Three things in dependency order: + +**16a — PostgreSQL + SQLAlchemy:** +- Users table: id, email, hashed_password, created_at +- Conversations table: id, user_id, created_at +- Messages table: id, conversation_id, role, content, metadata, created_at +- Alembic for migrations +- Add `postgres` service to docker-compose.yml + +**16b — OAuth2 + JWT:** +- Fill in `auth/oauth.py` placeholder +- `POST /auth/register` and `POST /auth/login` endpoints +- JWT tokens, `/api/query` requires valid token +- `user_id` extracted from token, replaces IP-based rate limiting + +**16c — Per-user history in frontend:** +- History sidebar in Streamlit +- `GET /api/conversations` endpoint +- `GET /api/conversations/{id}/messages` endpoint + +### Phase 17 — Grafana + Prometheus Monitoring +- FastAPI exports metrics via `prometheus-fastapi-instrumentator` +- Prometheus scrapes metrics every 15s +- Grafana dashboard showing: + - Queries per hour + - Average confidence score over time + - Cost per day + - Blocked query rate + - P95 response latency +- Add `prometheus` and `grafana` services to docker-compose.yml + +--- + +## How to Run Locally +```bash +# terminal 1 — backend +cd VaultMind/backend +uvicorn api.main:app --host 127.0.0.1 --port 8000 + +# terminal 2 — frontend +cd VaultMind/frontend +streamlit run streamlit_app.py + +# or with Docker +cd VaultMind +docker compose up +``` + +## Environment Variables needed in .env +``` +OPENAI_API_KEY= +LANGCHAIN_API_KEY= +LANGCHAIN_TRACING_V2=true +LANGCHAIN_PROJECT=vaultmind +``` \ No newline at end of file diff --git a/backend/api/main.py b/backend/api/main.py index 74e1ea7..7ad26cd 100644 --- a/backend/api/main.py +++ b/backend/api/main.py @@ -8,7 +8,7 @@ from api.middleware.rate_limit import add_rate_limit_middleware from api.routes import health, query -# ── Logging ─────────────────────────────────────────────────────────────────── +# -- Logging -- # One logger for the whole API layer. # Format: timestamp | level | logger name | message @@ -21,7 +21,7 @@ @asynccontextmanager async def lifespan(app:FastAPI): - # ── STARTUP ── + # -- STARTUP -- logger.info("VaultMind backend starting up...") logger.info("Compiling LangGraph pipeline...") @@ -32,7 +32,7 @@ async def lifespan(app:FastAPI): yield # ← server runs here, handling requests - # ── SHUTDOWN ── + # -- SHUTDOWN -- logger.info("VaultMind backend shutting down. Goodbye.") app = FastAPI( @@ -60,11 +60,9 @@ async def lifespan(app:FastAPI): ) # rate limit middleware - add_rate_limit_middleware(app) # Routers - app.include_router(health.router, prefix="",tags=["Health"]) app.include_router(query.router, prefix="/api",tags=["Query"]) diff --git a/backend/api/middleware/rate_limit.py b/backend/api/middleware/rate_limit.py index 07c3fa6..f8d234c 100644 --- a/backend/api/middleware/rate_limit.py +++ b/backend/api/middleware/rate_limit.py @@ -1,13 +1,7 @@ """ -backend/api/middleware/rate_limit.py - Rate limiting for the VaultMind API. Protects the backend from abuse and runaway costs, every query hits OpenAI, so an unbounded API is a credit card risk, not just a performance risk. - -Current strategy : 3 requests / minute / IP (in-memory) -Phase 15 upgrade : swap MemoryStorage → RedisStorage, everything else stays -Phase 16 upgrade : swap IP key → user_id from JWT, one line change """ from slowapi import Limiter diff --git a/backend/api/routes/query.py b/backend/api/routes/query.py index f22a2d1..5404b3c 100644 --- a/backend/api/routes/query.py +++ b/backend/api/routes/query.py @@ -53,7 +53,6 @@ class QueryResponse(BaseModel): retrieval_status: str = "" retrieved_chunks: int = 0 # count only, not the raw text — keeps response small - # -- Endpoint -- @router.post( "/query", diff --git a/k8s/backend-deployment.yaml b/k8s/backend-deployment.yaml new file mode 100644 index 0000000..d1a4b73 --- /dev/null +++ b/k8s/backend-deployment.yaml @@ -0,0 +1,85 @@ +# k8s/backend-deployment.yaml +# A Deployment manages our backend pods. +# It ensures the desired number of replicas is always running, +# restarts crashed pods, and handles rolling updates when we push new images. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vaultmind-backend + namespace: vaultmind +spec: + # 1 replica only — ChromaDB and chunks.json live on disk inside this pod. + # Running 2+ replicas would give each pod separate data copies. + # Phase 15 (Pinecone) removes this constraint by moving to a shared external vector DB. + replicas: 1 + selector: + matchLabels: + app: vaultmind-backend + template: + metadata: + labels: + app: vaultmind-backend + spec: + containers: + - name: vaultmind-backend + image: ghcr.io/devdoshi19/vaultmind-backend:latest + ports: + - containerPort: 8000 + imagePullPolicy: Always + + env: + - name: OPENAI_API_KEY + valueFrom: + secretKeyRef: + name: vaultmind-secret + key: OPENAI_API_KEY + - name: LANGCHAIN_API_KEY + valueFrom: + secretKeyRef: + name: vaultmind-secret + key: LANGCHAIN_API_KEY + + envFrom: + - configMapRef: + name: vaultmind-config + + # Mount the PVC at two paths inside the container — + # same paths the app expects based on config.py. + # This replaces the Docker Compose volume mounts. + volumeMounts: + - name: backend-storage + mountPath: /app/chroma_db + subPath: chroma_db + - name: backend-storage + mountPath: /app/data + subPath: data + + readinessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: 15 + periodSeconds: 10 + + livenessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: 30 + periodSeconds: 20 + failureThreshold: 3 + + resources: + requests: + memory: "512Mi" + cpu: "250m" + limits: + memory: "1Gi" + cpu: "500m" + + # Declare the PVC as a volume the pod can use. + # Both volumeMounts above reference this single volume by name. + volumes: + - name: backend-storage + persistentVolumeClaim: + claimName: vaultmind-backend-pvc \ No newline at end of file diff --git a/k8s/backend-pvc.yaml b/k8s/backend-pvc.yaml new file mode 100644 index 0000000..7d6109a --- /dev/null +++ b/k8s/backend-pvc.yaml @@ -0,0 +1,16 @@ +# k8s/backend-pvc.yaml +# A PersistentVolumeClaim requests disk storage from the cluster. +# minikube auto-provisions the actual volume behind this claim. +# This gives ChromaDB a real persistent disk that survives pod restarts — +# the same role that Docker Compose volumes played locally. +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: vaultmind-backend-pvc + namespace: vaultmind +spec: + accessModes: + - ReadWriteOnce # one pod can read/write at a time (fine since replicas=1) + resources: + requests: + storage: 1Gi # 1GB is plenty for ChromaDB + chunks.json \ No newline at end of file diff --git a/k8s/backend-service.yaml b/k8s/backend-service.yaml new file mode 100644 index 0000000..67156be --- /dev/null +++ b/k8s/backend-service.yaml @@ -0,0 +1,20 @@ +# k8s/backend-service.yaml +# A Service gives the backend a stable internal DNS name. +# Without this, the frontend would need the pod's IP — which changes on every restart. +# ClusterIP means this service is only reachable from inside the cluster. +# The frontend calls http://vaultmind-backend-service:8000 and K8s routes it here. +apiVersion: v1 +kind: Service +metadata: + name: vaultmind-backend-service + namespace: vaultmind +spec: + type: ClusterIP + # selector tells the Service which pods to route traffic to. + # Must match the labels on the backend Deployment's pod template. + selector: + app: vaultmind-backend + ports: + - protocol: TCP + port: 8000 # port the Service exposes inside the cluster + targetPort: 8000 # port the container is actually listening on \ No newline at end of file diff --git a/k8s/configmap.yaml b/k8s/configmap.yaml new file mode 100644 index 0000000..8aa97ea --- /dev/null +++ b/k8s/configmap.yaml @@ -0,0 +1,19 @@ +# k8s/configmap.yaml +# ConfigMap holds non-sensitive environment variables. +# Unlike Secrets, these are plain text and safe to commit. +# Pods reference this by name — if you change a value here, +# you re-apply this file and restart the pods. No image rebuild needed. +apiVersion: v1 +kind: ConfigMap +metadata: + name: vaultmind-config + namespace: vaultmind +data: + # LangSmith tracing config + LANGCHAIN_TRACING_V2: "true" + LANGCHAIN_PROJECT: "vaultmind" + + # Frontend uses this to reach the backend inside the cluster. + # "vaultmind-backend-service" is the name of the backend Service we'll create next. + # K8s internal DNS resolves service names automatically within the same namespace. + BACKEND_URL: "http://vaultmind-backend-service:8000" \ No newline at end of file diff --git a/k8s/frontend-deployment.yaml b/k8s/frontend-deployment.yaml new file mode 100644 index 0000000..ad78eb1 --- /dev/null +++ b/k8s/frontend-deployment.yaml @@ -0,0 +1,58 @@ +# k8s/frontend-deployment.yaml +# Manages the Streamlit frontend pod. +# The frontend is stateless — it only calls the backend over HTTP. +# No disk-bound state means this could scale to multiple replicas, +# but 1 is enough for a personal project. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vaultmind-frontend + namespace: vaultmind +spec: + replicas: 1 + selector: + matchLabels: + app: vaultmind-frontend + template: + metadata: + labels: + app: vaultmind-frontend + spec: + containers: + - name: vaultmind-frontend + image: ghcr.io/devdoshi19/vaultmind-frontend:latest + ports: + - containerPort: 8501 + imagePullPolicy: Always + + # BACKEND_URL comes from ConfigMap — "http://vaultmind-backend-service:8000" + # K8s internal DNS resolves this to the backend Service automatically. + # This is the K8s equivalent of Docker Compose's service name resolution. + envFrom: + - configMapRef: + name: vaultmind-config + + # Readiness probe — wait for Streamlit to finish booting before sending traffic. + readinessProbe: + httpGet: + path: / + port: 8501 + initialDelaySeconds: 10 + periodSeconds: 10 + + # Liveness probe — restart the pod if Streamlit becomes unresponsive. + livenessProbe: + httpGet: + path: / + port: 8501 + initialDelaySeconds: 20 + periodSeconds: 20 + failureThreshold: 3 + + resources: + requests: + memory: "256Mi" + cpu: "125m" + limits: + memory: "512Mi" + cpu: "250m" \ No newline at end of file diff --git a/k8s/frontend-service.yaml b/k8s/frontend-service.yaml new file mode 100644 index 0000000..09befa9 --- /dev/null +++ b/k8s/frontend-service.yaml @@ -0,0 +1,19 @@ +# k8s/frontend-service.yaml +# NodePort exposes the frontend outside the cluster. +# K8s maps nodePort 30501 on the minikube node to port 8501 in the container. +# We use minikube service command to get the browser URL — minikube +# handles the tunneling from your Windows machine into the cluster. +apiVersion: v1 +kind: Service +metadata: + name: vaultmind-frontend-service + namespace: vaultmind +spec: + type: NodePort + selector: + app: vaultmind-frontend + ports: + - protocol: TCP + port: 8501 # port exposed inside the cluster + targetPort: 8501 # port Streamlit listens on inside the container + nodePort: 30501 # port opened on the minikube node (30000-32767 range) \ No newline at end of file diff --git a/k8s/namespace.yaml b/k8s/namespace.yaml new file mode 100644 index 0000000..19a63f8 --- /dev/null +++ b/k8s/namespace.yaml @@ -0,0 +1,8 @@ +# k8s/namespace.yaml +# A namespace is a logical boundary inside the K8s cluster. +# All VaultMind resources (pods, services, secrets) live here. +# This keeps our stuff isolated from other apps on the same cluster. +apiVersion: v1 +kind: Namespace +metadata: + name: vaultmind \ No newline at end of file