A Rust job queue for LLM inference workloads. PostgreSQL-backed with LISTEN/NOTIFY for worker dispatch.
Three concurrent tokio tasks:
- HTTP API — Axum server with POST/GET endpoints for job submission and status
- Scheduler — polls for due Pending jobs, promotes to Running, NOTIFYs workers
- Heartbeat Monitor — detects timed-out Running jobs, marks Failed with retry logic
Workers listen on a Postgres channel and execute jobs concurrently via tokio::sync::Semaphore.
Pending → Running → Completed
→ Failed → retry_count+1 → if retries left → Pending
→ if exhausted → PermanentlyFailed → DLQ
# Set up database
createdb job_queue
sqlx migrate run
# Run the server
WORKER_CONCURRENCY=10 cargo run
# Submit a job
curl -X POST http://localhost:3000/jobs \
-H "Content-Type: application/json" \
-d '{"prompt": "Hello, world"}'
# Check job status
curl http://localhost:3000/jobs/<job_id>
# Run load test
cargo run -p load-tester -- --concurrency 20 --total-jobs 200| Env Var | Default | Description |
|---|---|---|
DATABASE_URL |
(required) | Postgres connection string |
WORKER_CONCURRENCY |
10 | Max concurrent jobs per worker |
HEARTBEAT_TIMEOUT |
30s | Seconds before a running job is considered dead |
- Rust with tokio async runtime
- Axum for HTTP
- SQLx with compile-time checked queries
- PostgreSQL with LISTEN/NOTIFY
- No managed queues, no Redis