Skip to content

Production-Ready Kubernetes Microservices, Observability, and High-Performance Async Architecture#5

Merged
bit-web24 merged 8 commits into
masterfrom
dev
Jun 21, 2026
Merged

Production-Ready Kubernetes Microservices, Observability, and High-Performance Async Architecture#5
bit-web24 merged 8 commits into
masterfrom
dev

Conversation

@bit-web24

Copy link
Copy Markdown
Owner

🚀 Overview

This major release merges the fully completed, production-ready dev branch into master. It transitions Bittuly from a set of isolated local applications into a highly resilient, observable, and scalable distributed microservice architecture deployed via Kubernetes.

The system has been heavily optimized with distributed caching, message brokering, and asynchronous eventing, and has been proven under extreme load testing to handle 6,500+ RPS with sub-millisecond median latencies.

🏗️ Architecture & Infrastructure Highlights

  • Kubernetes Orchestration: Implemented complete K8s manifests (Deployments, Services, ConfigMaps, Secrets) across all services.
  • Horizontal Pod Autoscaling (HPA): Configured automatic CPU-based scaling for microservices to dynamically handle viral traffic spikes.
  • NGINX Ingress Controller: Configured robust API routing, rate-limiting, and CORS handling at the edge.
  • CI/CD Pipelines: Integrated GitHub actions for automated Rust formatting, linting (clippy), dependency security auditing, and running isolated sqlx integration tests against ephemeral Docker databases.

⚡ Performance & Caching Optimizations

  • L1 & L2 Caching Architecture: Implemented a two-tier caching strategy in url-service.
    • L2 (Redis): Distributed cache for sub-millisecond redirect lookups, protecting PostgreSQL.
    • L1 (Moka): In-memory microcache utilizing "Singleflight Coalescing" to entirely prevent Thundering Herd bottlenecks.
  • Asynchronous RabbitMQ Eventing: Decoupled the analytics/click-tracking pipeline from the hot HTTP request path using tokio::spawn, preventing thread starvation and connection pool exhaustion under heavy load.
  • Load Testing Validated: Achieved 0.00% error rates and 3.62ms p(99) latency at ~7,000 requests per second during sustained k6 cluster testing.

🔭 Observability & Telemetry Stack

  • Prometheus & Grafana: Embedded Prometheus metrics endpoints natively into the Rust Axum routers for real-time monitoring of cache hits, queue sizes, and HTTP response times.
  • OpenTelemetry Distributed Tracing: Integrated end-to-end distributed tracing via Jaeger. Spans are now correctly propagated across HTTP boundaries and AMQP queues (from url-service directly to consumer-service).

📦 Services Included

  1. Auth Service (auth-service): JWT-based authentication, user registration, and secure credential management.
  2. URL Service (url-service): The high-throughput redirect engine and link generation API.
  3. Consumer Service (consumer-service): Background worker processing RabbitMQ queues for analytics aggregation and Dead Letter Queue (DLQ) retry logic.
  4. Frontend (frontend-service): React/Vite web application served efficiently via NGINX.

- Fix Axum router fallback bug returning 401 instead of 404
- Update auth-service and url-service Dockerfiles to resolve glibc mismatch
- Create web/Dockerfile and web/nginx.conf to containerize the Vite frontend
- Configure frontend NGINX for custom fallback routing (intercept 404s and proxy to url-service or serve React SPA)
- Add K8s Deployment and Service manifests for frontend-service
- Update NGINX Ingress Controller routing to send root traffic to frontend-service
- Expand scripts/deploy-local.sh to build and deploy frontend
- Update TARGET.md to mark Phase 7 as completed
- Add Kubernetes deployment instructions to README.md
- Start postgres-auth, postgres-urls, redis, and rabbitmq via docker-compose
  for local test runs (only kind was running, causing PoolTimedOut)
- auth-service tests require DATABASE_URL=postgres://...@localhost:5432/bittuly_auth
- url-service tests require DATABASE_URL=postgres://...@localhost:5433/bittuly_urls
  (handler tests also need Redis at localhost:6379 for UrlState setup)
- Add scripts/test.sh: smart runner that auto-starts infra and sets correct
  DATABASE_URL per service — run ./scripts/test.sh to test everything
- Update .cargo/config.toml: use force=false so DATABASE_URL can be overridden
  from the shell, and add test-auth/test-urls aliases
- All 26 tests now pass (13 auth + 13 url)
Mirrors the docker compose up/down UX for the kind cluster:

  ./scripts/k8s.sh up        - create cluster + full deploy (first time)
  ./scripts/k8s.sh start     - resume a stopped cluster, no rebuild
  ./scripts/k8s.sh stop      - pause containers, data preserved
  ./scripts/k8s.sh down      - delete cluster, images kept (fast redeploy)
  ./scripts/k8s.sh purge     - delete cluster + all bittuly Docker images
  ./scripts/k8s.sh deploy    - rebuild images + rolling redeploy
  ./scripts/k8s.sh status    - show nodes, pods, services
  ./scripts/k8s.sh logs <s>  - tail pod logs for a service
  ./scripts/k8s.sh restart <s> - rolling restart a deployment
  ./scripts/k8s.sh jaeger    - port-forward Jaeger UI → :16686
  ./scripts/k8s.sh rabbitmq  - port-forward RabbitMQ UI → :15672
ci.yml — fixed and expanded:
- Split into 5 parallel jobs: rust-lint, test-auth, test-url, docker-build, frontend
- Add postgres-auth service container for auth-service tests (DATABASE_URL fixed)
- Add postgres-urls + redis service containers for url-service tests
- Add cargo-audit security scanning
- Use docker/build-push-action with GHA layer cache for fast Docker smoke builds
- Fix Node.js version from 26 to 20 (LTS)
- tests now run in parallel (test-auth and test-url run concurrently)

cd.yml — new:
- Triggers on every push to main (after CI passes)
- Builds and pushes all 4 images to GHCR tagged with git SHA + latest
- Uses docker layer cache (type=gha) for fast incremental builds
- Deploy step scaffolded with both kubectl and ArgoCD options (commented,
  ready to enable when production cluster is provisioned)
url-service/handlers.rs:
- Remove unused imports: moka::future::Cache, std::time::Duration
- Remove mut from 4 test response variables (unused_mut)

url-service/services.rs:
- Remove unused DateTime from chrono import in test module

url-service/mock.rs:
- Collapse nested if-let blocks (clippy::collapsible_if)
  - add_shorten_url: merge expires_at check into single if-let && guard
  - delete_url: merge user_id check into single if-let && guard

auth-service/services.rs:
- Remove unused imports: async_trait, HashMap, RwLock from test module

auth-service/handlers.rs:
- Remove mut from 3 test response variables (unused_mut)

All 16 clippy errors resolved. cargo clippy --workspace --all-targets -- -D warnings now passes clean.
The jsonwebtoken crate depends on rsa, which has an unfixable timing
side-channel vulnerability (Marvin Attack). Since we only use HS256
(symmetric HMAC) for our JWTs and do not use RSA keys, we are
unaffected by this vulnerability.

- Added --ignore RUSTSEC-2023-0071 to ci.yml
- Added audit.toml for local cargo audit runs
@bit-web24
bit-web24 merged commit fc3308f into master Jun 21, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant