[MOSIP Nexus] Deploy MOSIP Nexus on Rancher — Lower Environment Setup

[mohanachandran-s]

Context

MOSIP Nexus is an AI-powered internal knowledge assistant for the MOSIP team.
This issue captures everything the DevOps team needs to deploy it on Rancher
in a lower environment.

Repo: https://github.com/your-org/mosip-labs
K8s manifests: Server/k8s/ (API + pipeline) and UI/k8s/ (React UI)
Point of contact: @MohanaChandran (engineering queries)

---

What Engineering Will Provide

Before deployment begins, the engineering team will hand over:

• Encrypted Server/k8s/02-sealed-secret.yaml (generated via seal-secrets.sh — safe to apply, do not use the plaintext 02-secret.yaml template with real values)
• Confirmation that Docker images are available at the registry

---

Prerequisites (one-time cluster setup)

Prerequisite	Why Needed	How to Install
cert-manager	Auto-issues TLS certificates via Let's Encrypt	Rancher UI → Apps → cert-manager
Sealed Secrets controller	Decrypts the SealedSecret in git	kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml
Rancher Monitoring (optional)	Enables Prometheus alert rules + HPA metrics	Rancher UI → Apps → Monitoring → Install
metrics-server	Required by HPA autoscaler	Included in RKE2 by default — verify with kubectl top pods

---

1. Docker Images

Component	Image	Pull Policy
API + pipeline	nexus-server:v1.0.0	IfNotPresent
UI (React/nginx)	nexus-ui:v1.0.0	IfNotPresent
Database	pgvector/pgvector:pg16	IfNotPresent

Bump the image tag on every release. Never use :latest in any K8s manifest.

---

2. Resource Requirements

Component	CPU Request	CPU Limit	Memory Request	Memory Limit
nexus-api	500m	2 cores	1500Mi	4Gi
nexus-ui	250m	1 core	256Mi	512Mi
nexus-postgres	250m	1 core	512Mi	2Gi

Note: The initial ingest Job (07-initial-ingest-job.yaml) crawls all knowledge sources and builds the vector database. It is CPU/memory intensive and takes 4–8 hours on first run. Resources above are for steady-state operation only.

Autoscaling: nexus-api is managed by an HPA (09-hpa.yaml) that scales between 1–4 replicas when CPU >70% or memory >80%.

---

3. Ports

Component	Container Port	Service Port	Protocol
nexus-api	8000	8000	HTTP
nexus-ui	80	80	HTTP
nexus-postgres	5432	5432	TCP (internal only)

---

4. URLs / Access Paths

	URL
Chat UI	https://mosip-nexus.env.mosip.net/
API	https://mosip-nexus.env.mosip.net/api/
API Docs	https://mosip-nexus.env.mosip.net/api/docs
API Health	https://mosip-nexus.env.mosip.net/api/health
API Metrics	https://mosip-nexus.env.mosip.net/api/metrics

Please confirm whether routing is path-based (/api/) or subdomain-based (mosip-nexus-api.env.mosip.net) and update Server/k8s/05-ingress-api.yaml accordingly.

---

5. Health Check Paths

Component	Path	Port	Initial Delay
nexus-api	GET /health	8000	60s
nexus-ui	GET /	80	30s
nexus-postgres	pg_isready -U mosip -d mosipnexus	5432	30s

---

6. Persistent Volumes Required

PVC Name	Mount Path	Size	Used By
nexus-data	/app/MosipNexus/data	10Gi	CronJob, ingest Job only
postgres-data	/var/lib/postgresql/data	20Gi	nexus-postgres
nexus-db-backups	/backups	20Gi	nexus-db-backup CronJob (daily pg_dump)

Set storageClassName in Server/k8s/01-postgres.yaml and Server/k8s/06-cronjob.yaml to match your Rancher storage class before applying.

Note: nexus-api does not mount nexus-data — it reads only from pgvector at runtime. Only the CronJob and ingest Job write to nexus-data, ensuring no ReadWriteOnce contention.

---

7. Environment Variables

🔴 Mandatory

Variable	Notes
GROQ_API_KEY	Provided by engineering
POSTGRES_PASSWORD	Set any strong password — must match PG_CONNECTION
PG_CONNECTION	postgresql+psycopg://mosip:<password>@nexus-postgres:5432/mosipnexus

🟡 Required for full knowledge base ingestion

These are needed by the initial ingest Job to crawl all knowledge sources. Without them, the job will skip those sources.

Variable	Notes
GITHUB_TOKEN	Provided by engineering — needed to crawl GitHub issues
CONFLUENCE_URL	https://your-org.atlassian.net/wiki
CONFLUENCE_USER	Atlassian login email — provided by engineering
CONFLUENCE_TOKEN	API token — provided by engineering
CONFLUENCE_SPACE_KEYS	QT,ENGG,PMS,MSD

🟡 Recommended (job failure alerts)

Without these, CronJob failures are silent — no one is notified if the nightly update fails.

Variable	Notes
SMTP_HOST	e.g. smtp.gmail.com
SMTP_PORT	e.g. 587
SMTP_USER	Sender email address
SMTP_PASS	Sender email password / app password
NOTIFY_EMAIL	Alert recipient email address

🟢 Optional (not needed for lower env)

JIRA_*, LANGCHAIN_*, HF_TOKEN

All variables are stored in Server/k8s/02-sealed-secret.yaml (encrypted). Engineering will provide this file — apply it directly, do not commit plaintext secrets to git.

---

8. Ingress Requirements

• Ingress controller: nginx
• TLS via cert-manager — requires a ClusterIssuer named letsencrypt-prod (template included as a comment in Server/k8s/05-ingress-api.yaml)
• Proxy timeout: 300s (AI responses can take up to 30s)
• SSL redirect: enabled (HTTP → HTTPS enforced)

---

9. Deployment Steps

# ── Prerequisites ──────────────────────────────────────────────────────────────
# Run seal-secrets.sh once to generate the encrypted secret from Server/.env
# (requires kubeseal CLI and Sealed Secrets controller installed in cluster)
bash Server/k8s/seal-secrets.sh
# → produces Server/k8s/02-sealed-secret.yaml

# ── Step 1: Core infrastructure ────────────────────────────────────────────────
kubectl apply -f Server/k8s/00-namespace.yaml
kubectl apply -f Server/k8s/01-postgres.yaml
kubectl apply -f Server/k8s/02-sealed-secret.yaml
kubectl apply -f Server/k8s/04-service-api.yaml

# ── Step 2: Monitoring (skip if Rancher Monitoring is not installed) ───────────
kubectl apply -f Server/k8s/10-monitoring.yaml

# ── Step 3: Verify postgres is ready before ingestion ─────────────────────────
kubectl get pods -n mosip-nexus

# ── Step 4: Run initial ingest job ─────────────────────────────────────────────
# Crawls all knowledge sources and builds the vector database.
# Expected runtime: 4–8 hours on first run.
kubectl apply -f Server/k8s/07-initial-ingest-job.yaml

# Monitor progress
kubectl logs -n mosip-nexus job/nexus-initial-ingest -f

# Wait for completion before proceeding
kubectl wait --for=condition=complete job/nexus-initial-ingest \
  -n mosip-nexus --timeout=7200s

# ── Step 5: API deployment + autoscaling ───────────────────────────────────────
kubectl apply -f Server/k8s/03-deployment-api.yaml
kubectl apply -f Server/k8s/09-hpa.yaml

# ── Step 6: UI deployment ──────────────────────────────────────────────────────
kubectl apply -f UI/k8s/01-deployment-ui.yaml
kubectl apply -f UI/k8s/02-service-ui.yaml

# ── Step 7: Ingress (TLS — cert-manager must be installed) ────────────────────
kubectl apply -f Server/k8s/05-ingress-api.yaml
kubectl apply -f UI/k8s/03-ingress-ui.yaml

# ── Step 8: CronJobs ───────────────────────────────────────────────────────────
kubectl apply -f Server/k8s/06-cronjob.yaml          # nightly knowledge update (03:00 UTC)
kubectl apply -f Server/k8s/08-postgres-backup.yaml  # daily pg_dump (03:30 UTC, 7-day retention)