AI Feature Router for Image Metadata

Local‑first image captioning + Hybrid Search (Vector + Keyword) with a scalable Go microservice for high-throughput retrieval. Built to demonstrate AI integration, polyglot architecture, and production‑grade engineering.

Why this project

Scalable Search: Dedicated Go microservice for high-performance hybrid search (pgvector + FTS).
Smart Routing: Local vs cloud policy routing for captioning (confidence & SLO‑based).
Production Engineering: Shadow Mode deployment, Load Testing (Locust), and ADRs.
Observability: Prometheus, Grafana, and Jaeger tracing across Python and Go services.
Multi-tenant: Supabase authentication with user-owned private/public images.

Quick Start

Option 1: Run API Locally

# Start infrastructure (Postgres, Qdrant, etc.)
docker compose -f infra/docker-compose.yml up -d

# Run API with uvicorn
uvicorn apps.api.main:app --host 0.0.0.0 --port 8000
# open http://localhost:8000/docs

Option 2: Run API in Docker

# 1. Create .env.docker file (see Configuration section below)
cp .env.example .env.docker
# Edit .env.docker with your values

# 2. Build API image with embedder support
docker build -f apps/api/Dockerfile --build-arg INSTALL_EMBEDDER=true -t imagesearch-api:embedder .

# 3. Start infrastructure
cd infra && docker-compose up -d

# 4. Run API container
docker run -d --name api \
  --network infra_default \
  -p 8000:8000 \
  --env-file .env.docker \
  imagesearch-api:embedder

# open http://localhost:8000/docs

Run Frontend (Next.js UI)

The project includes a full-featured web UI in apps/ui/:

cd apps/ui
cp .env.local.example .env.local
# Edit .env.local with your Supabase credentials and API URL
npm install
npm run dev
# open http://localhost:3100

See the Frontend section below for detailed setup and features.

Run Mobile Companion (React Native / Expo)

The project also includes a mobile companion app in apps/mobile/:

cd apps/mobile
cp .env.example .env
# Edit .env with the API URL and Supabase publishable config
npm install
npx expo start

See the React Native Companion App section and mobile reviewer guide for screenshots, architecture, demo script, and manual QA.

Storage Options

Local - Files stored on disk (default, zero setup)
MinIO - S3-compatible, runs in Docker (dev/self-hosted)
Cloudflare R2 - Zero egress fees, $1.50/month for 100GB (production)
AWS S3 - Industry standard with CloudFront CDN

See S3_STORAGE_SETUP.md for complete setup guide.

Architecture

FastAPI gateway: Handles writes, auth, and orchestrates requests.
Go Search Service: Dedicated microservice for high-performance read-only search.
Authentication: Supabase JWT-based auth with user profiles.
Multi-tenant: User-owned images with privacy controls (private/public).
Captioner: local BLIP first; cloud fallback (OpenAI/Gemini/Anthropic) via adapter.
Embedder: OpenCLIP (or SigLIP) image+text encoders → vectors.
Vector store: pgvector (HNSW) or Qdrant, selectable via env.

[Client/UI] → Next.js (Supabase Auth) → FastAPI (Gateway)
                                      → (caption: BLIP → cloud?)
                                      → (embed: OpenCLIP)
                                      → Go Search Service (Read Path)
                                      → Vector Store (pgvector/Qdrant)
                                      → Prometheus/Jaeger → Grafana

## Go Search Service (New)
To improve scalability and performance for read-heavy search traffic, we have introduced a dedicated **Go Search Service**.
- **Role**: Handles `POST /search` requests (hybrid vector + keyword search).
- **Stack**: Go 1.23, `pgx`, `pgvector`.
- **Performance**: Lower tail latency (P99) and better resource efficiency than the Python baseline.
- **Integration**: The Python API proxies search requests to this service when `SEARCH_BACKEND=go` is set.

See [Scaling Go Search Service](docs/scaling_go_search.md) for detailed benchmarks.

Endpoints

Public Endpoints

GET /search?q=... – semantic text→image search; query params: k, scope (all/mine/public)
GET /images – list images with filtering; query params: limit, offset, visibility
GET /images/{id} – metadata including download URLs (respects privacy)
GET /images/{id}/download – download original image (respects privacy)
GET /images/{id}/thumbnail – download 256x256 thumbnail (respects privacy)
GET /metrics – Prometheus exposition
GET /health – health check

Authenticated Endpoints (requires Bearer token)

POST /images – upload image (URL or file); returns {id, caption, download_url, thumbnail_url, ...}
PATCH /images/{id} – update image metadata (caption, visibility)
DELETE /images/{id} – soft delete image (owner only)

Configuration

Local Development (.env)

For running the API with uvicorn locally:

# Vector Store
VECTOR_BACKEND=pgvector
DATABASE_URL=postgresql+psycopg://postgres:postgres@localhost:5432/ai_router
QDRANT_URL=http://localhost:6333

# Supabase Authentication
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_JWT_SECRET=your-jwt-secret-for-hs256-projects
SUPABASE_SECRET_KEY=sb_secret_your-secret-key
ADMIN_USER_ID=your-admin-user-uuid

# Cloud providers
CLOUD_PROVIDER=openrouter
OPENROUTER_API_KEY=sk-or-v1-...
OPENROUTER_MODEL=openai/gpt-4o-mini

# Routing policy
CAPTION_CONFIDENCE_THRESHOLD=0.55
CAPTION_LATENCY_BUDGET_MS=600

# Model Configuration
USE_MOCK_MODELS=false

# Image storage
IMAGE_STORAGE_BACKEND=local
IMAGE_STORAGE_PATH=./storage/images
BASE_URL=http://localhost:8000

The API verifies user access tokens with Supabase Auth. Legacy/shared-secret HS256 tokens use SUPABASE_JWT_SECRET; asymmetric ES256/RS256 tokens are verified from the project JWKS endpoint under SUPABASE_URL.

Docker Deployment (.env.docker)

For running the API in Docker, create .env.docker with Docker network hostnames:

# Vector Store Configuration (Docker network hostnames)
VECTOR_BACKEND=pgvector
DATABASE_URL=postgresql+psycopg://postgres:postgres@infra-postgres-1:5432/ai_router
QDRANT_URL=http://infra-qdrant-1:6333

# Model Configuration
# Set to false to use real models (requires embedder image)
USE_MOCK_MODELS=false

# Cloud Provider Configuration
CLOUD_PROVIDER=openrouter
OPENROUTER_API_KEY=sk-or-v1-your-key-here
OPENROUTER_MODEL=openai/gpt-4o-mini

# Routing Policy
CAPTION_CONFIDENCE_THRESHOLD=0.55  # Use 0.99 to force cloud usage
CAPTION_LATENCY_BUDGET_MS=600

# Cloud Rate Limiting
CLOUD_MAX_REQUESTS_PER_MINUTE=60
CLOUD_MAX_REQUESTS_PER_DAY=10000
CLOUD_DAILY_BUDGET_USD=10.00

# Supabase Configuration (Multi-tenant Authentication)
# Get these from: https://supabase.com/dashboard/project/YOUR_PROJECT/settings/api
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_JWT_SECRET=your-jwt-secret-for-hs256-projects
SUPABASE_SECRET_KEY=sb_secret_your-secret-key
ADMIN_USER_ID=your-admin-user-uuid

# Image Storage (Docker)
IMAGE_STORAGE_BACKEND=minio  # or local, s3
IMAGE_STORAGE_PATH=/app/storage/images
BASE_URL=http://localhost:8000

# MinIO Configuration (if using minio backend)
S3_BUCKET_NAME=imagesearch
S3_ENDPOINT_URL=http://infra-minio-1:9000
S3_ACCESS_KEY_ID=minioadmin
S3_SECRET_ACCESS_KEY=minioadmin
S3_REGION=us-east-1
S3_USE_PRESIGNED_URLS=true
S3_PRESIGNED_URL_EXPIRY=3600

Key differences for Docker:

Use Docker service names (e.g., infra-postgres-1 instead of localhost)
Set USE_MOCK_MODELS=false if you built with INSTALL_EMBEDDER=true
Use minio storage backend with Docker service name infra-minio-1
Get Supabase credentials from your project's API settings page

Multi-Tenant Setup

1. Create Supabase Project

Go to supabase.com and create a new project
Get your credentials from Settings → API:
- Project URL
- Publishable key (sb_publishable_...) for client-side UI
- Secret key (sb_secret_...) for backend/admin operations only; never expose this in browser code
- JWT Secret (Settings → API → JWT Settings)

2. Run Database Migrations

# Connect to your database and run migrations
psql $DATABASE_URL -f apps/api/storage/migrations/001_add_profiles.sql
psql $DATABASE_URL -f apps/api/storage/migrations/002_add_multi_tenant_fields.sql

3. Migrate Existing Data (Optional)

If you have existing images, migrate them to the admin user:

# Set environment variables
export DATABASE_URL="your-database-url"
export ADMIN_USER_ID="your-admin-uuid"

# Run migration
python scripts/migrate_to_multitenant.py --yes

4. Configure Environment

Update your .env file with Supabase credentials (see Config section above).

5. Configure Supabase

In your Supabase dashboard:

Authentication → URL Configuration: Add your frontend URL to redirect URLs
Authentication → Providers: Enable Email provider
Authentication → Email Templates: Customize if needed

See docs/multi_tenant/DEPLOYMENT_CHECKLIST.md for complete deployment guide.

Run tests

# Run all CI/CD safe tests
python tests/test_infrastructure.py
python tests/test_load.py
python tests/test_image_storage.py

# Multi-tenant E2E tests
pytest tests/test_e2e_multitenant.py -v

# Or use pytest for all tests
pytest tests/ -v

# See tests/README.md for more options

Dataset Seeding

Seed the database with real image datasets. Requires authentication since the /images endpoint is protected.

Get Authentication Token

Log in to your frontend (http://localhost:3100)
Open browser DevTools (F12) → Console
Run: JSON.parse(localStorage.getItem('sb-YOUR_PROJECT_ID-auth-token')).access_token
Copy the JWT token

Seed Images

# COCO validation set (automatic download)
python scripts/seed_datasets.py \
  --dataset coco \
  --count 100 \
  --auth-token "YOUR_JWT_TOKEN" \
  --visibility public

# Unsplash (requires API key)
python scripts/seed_datasets.py \
  --dataset unsplash \
  --count 50 \
  --api-key YOUR_UNSPLASH_KEY \
  --auth-token "YOUR_JWT_TOKEN" \
  --visibility public

# List available datasets
python scripts/seed_datasets.py --list

Note: Images are processed sequentially to avoid overwhelming cloud APIs. Expect ~10-15 seconds per image with cloud captioning.

Benchmarking

Run comprehensive performance benchmarks:

cd notebooks
python benchmark.py --test-image ../test_image.jpg --sample-size 100

Generates:

Latency statistics (mean, P95, P99)
Cost projections
Quality metrics (BLEU, METEOR)
Visualization plots

Observability

Prometheus at http://localhost:9090
Grafana at http://localhost:3000 (import infra/grafana/dashboards.json)
Jaeger at http://localhost:16686

ADRs

0001-routing-policy.md: Local-first with Confidence & SLO Overrides
0002-python-to-go-migration.md: Migrating search path to Go
0003-hybrid-search-implementation.md: Hybrid Search with pgvector + FTS

Frontend (Next.js UI)

Path: apps/ui/

Stack: Next.js 16 (App Router), React 18, TypeScript, Tailwind CSS, TanStack Query, Supabase Auth
Features: User authentication, Search gallery, Upload (URL or file) with progress, Image library, Privacy controls, Explore public images
Default port: 3100

Prerequisites

Node.js 18+ (recommended 20+)
Backend API running (default http://localhost:8000)
Supabase project (for authentication)

Setup & Run (Dev)

cd apps/ui
cp .env.local.example .env.local
# Edit .env.local with your values:
# NEXT_PUBLIC_API_BASE=http://localhost:8000
# NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
# NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY=sb_publishable_your-key
npm install
npm run dev   # http://localhost:3100

Build & Start (Prod)

cd apps/ui
npm run build
npm run start   # http://localhost:3100

Important environment

NEXT_PUBLIC_API_BASE – Backend API URL (e.g., http://localhost:8000)
NEXT_PUBLIC_SUPABASE_URL – Your Supabase project URL
NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY – Supabase publishable key (sb_publishable_...), safe for client-side use

UI routes

/ – Search page (query by text, shows local/cloud origin badges)
/login – User login with Supabase
/signup – User registration
/upload – Upload images (authenticated); set privacy (private/public)
/library – User's uploaded images with privacy controls
/explore – Browse public images from all users
/image/[id] – Image detail (caption, origin, metadata, "Find similar")
/metrics – Metrics dashboard (p50/p95, local vs cloud split, cost, cache hits)

API proxy routes (UI → Backend)

All proxy routes automatically forward the Authorization header from Supabase:

GET /api/search → ${NEXT_PUBLIC_API_BASE}/search (with auth token)
GET /api/images → ${NEXT_PUBLIC_API_BASE}/images (with auth token)
GET /api/images/[id] → ${NEXT_PUBLIC_API_BASE}/images/{id} (with auth token)
POST /api/images → ${NEXT_PUBLIC_API_BASE}/images (authenticated, file or url)
PATCH /api/images/[id] → ${NEXT_PUBLIC_API_BASE}/images/{id} (authenticated)
DELETE /api/images/[id] → ${NEXT_PUBLIC_API_BASE}/images/{id} (authenticated)
GET /api/metrics/summary → parses ${NEXT_PUBLIC_API_BASE}/metrics into a compact JSON summary

Troubleshooting

Hydration warnings from extensions: we set suppressHydrationWarning on <body>.

Dev over LAN (192.168.x.x) may show an "allowedDevOrigins" warning; add to apps/ui/next.config.mjs if needed:

const nextConfig = {
  images: { unoptimized: true },
  allowedDevOrigins: ['http://192.168.70.10:3100']
}
export default nextConfig

Tailwind warning about @tailwindcss/line-clamp: remove the plugin from tailwind.config.ts (included by default in v3.3+).

React Native Companion App

Path: apps/mobile/

The Expo app is a mobile-first client for the same FastAPI backend used by the web UI. It demonstrates native mobile product concerns around image capture, auth, offline state, async job tracking, and private/public image management while leaving AI routing and retrieval on the backend.

Core mobile flows:

Supabase sign-in and sign-up.
Public, mine, and all search scopes.
Camera/photo-library image upload through POST /images/async?priority=normal.
Private/public visibility selection.
Async ingestion job list and job detail polling.
Retry-pending local jobs for failed/offline uploads.
Owner library filters, visibility updates, soft delete, and image detail.
Offline banner, paused polling, stale cached search messaging, and local cache/queue cleanup.

Run it locally:

cd apps/mobile
cp .env.example .env
npm install
npx expo start

Mobile-safe env values:

EXPO_PUBLIC_API_BASE_URL=http://localhost:8000
EXPO_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
EXPO_PUBLIC_SUPABASE_PUBLISHABLE_KEY=your-supabase-publishable-or-anon-key

Use a LAN API URL for physical devices. Use http://10.0.2.2:8000 for Android Emulator.

Reviewer resources:

Production Deployment (Hybrid Architecture)

We use a Hybrid Deployment strategy to balance scalability and cost:

API & Search Service: Google Cloud Run (Serverless, scales to zero).
Worker & Redis: Google Compute Engine (VM, cost-effective for always-on background tasks).

See docs/architecture.md for the detailed architecture diagram.

1. Prerequisites

Google Cloud Project with Billing enabled.
gcloud CLI installed and authenticated.
.env.production file with production secrets.

2. Deploy Worker VM (Redis + Ingestion Worker)

This script provisions a small VM (e2-small), installs Docker, and deploys Redis and the Ingestion Worker.

./scripts/deploy_worker.sh

Note: This script outputs the VM's public IP and Redis password. Ensure these are updated in your .env.production.

3. Deploy Cloud Run Services (API + Go Search)

This script builds and deploys the stateless services to Cloud Run, connecting them to the Worker VM's Redis.

./deploy.sh

4. Frontend (Vercel)

Deploy the Next.js frontend to Vercel:

cd apps/ui
vercel --prod

Ensure you set the NEXT_PUBLIC_API_BASE environment variable in Vercel to your Cloud Run API URL.

See docs/multi_tenant/DEPLOYMENT_CHECKLIST.md for a complete checklist.

License

MIT (sample) – replace or update as you prefer.

Name		Name	Last commit message	Last commit date
apps		apps
costs		costs
data/coco		data/coco
docs		docs
infra		infra
notebooks		notebooks
scripts		scripts
services/search-go		services/search-go
tests		tests
workers		workers
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
cloudbuild-agent.yaml		cloudbuild-agent.yaml
cloudbuild-api-embedder.yaml		cloudbuild-api-embedder.yaml
cloudbuild-api.yaml		cloudbuild-api.yaml
cloudbuild-go-search.yaml		cloudbuild-go-search.yaml
deploy.sh		deploy.sh
pytest.ini		pytest.ini
test_image.jpg		test_image.jpg

Latest commit

History

Folders and files

Repository files navigation