Skip to content

ARCHITECTURE_DIAGRAM.md

Latest commit

 

History

History
131 lines (99 loc) · 4.04 KB

ARCHITECTURE_DIAGRAM.md

File metadata and controls

131 lines (99 loc) · 4.04 KB
Raw

Architecture Diagrams

Mermaid source. GitHub renders these natively in Markdown — no image files needed. For the PDF, paste into https://mermaid.live and export PNG.

1. System architecture

flowchart TD
    web["<b>apps/web</b><br/><span style='font-size:12px'>React 19 · polls every 2s</span>"]

    subgraph services["long-running services"]
        direction LR
        api["<b>apps/api</b><br/><span style='font-size:12px'>Express 5 · :3000</span>"]
        worker["<b>apps/worker</b><br/><span style='font-size:12px'>poll 500ms · scales out</span>"]
        sched["<b>apps/scheduler</b><br/><span style='font-size:12px'>3 loops · 10s · no leader</span>"]
    end

    pg[("<b>PostgreSQL 17</b><br/><span style='font-size:12px'>queue + metadata · single source of truth</span>")]
    ext["<b>external http</b><br/><span style='font-size:12px'>SSRF-allowlisted</span>"]

    web -- "REST + JWT" --> api
    api --> pg
    worker --> pg
    sched --> pg
    worker -.-> ext

    classDef client fill:#EDEAE3,stroke:#73726C,stroke-width:1.5px,rx:8,color:#2C2C2A
    classDef svc fill:#DCE9F7,stroke:#4A6E96,stroke-width:1.5px,rx:8,color:#1B3A5C
    classDef store fill:#E4E0F4,stroke:#6B5CA5,stroke-width:1.5px,rx:8,color:#3B2E6E
    classDef outside fill:#F7E4DC,stroke:#A5674A,stroke-width:1.5px,rx:8,color:#6E3B2E

    class web client
    class api,worker,sched svc
    class pg store
    class ext outside
Loading

2. Job lifecycle state machine

stateDiagram-v2
    [*] --> queued: enqueue (due now)
    [*] --> scheduled: enqueue (delayed / cron)

    scheduled --> queued: promoter — run_at <= now()<br/>FOR UPDATE SKIP LOCKED
    queued --> claimed: claim — FOR UPDATE SKIP LOCKED
    claimed --> running: handler starts

    running --> completed: success
    running --> failed: handler error
    running --> queued: SIGTERM drain<br/>(attempt NOT charged)

    failed --> scheduled: attempt < max<br/>backoff + full jitter
    failed --> dead_letter: attempt >= max

    claimed --> queued: lease expired<br/>(attempt refunded)
    running --> queued: lease expired<br/>(attempt refunded)

    queued --> cancelled: manual
    scheduled --> cancelled: manual
    dead_letter --> queued: DLQ replay

    completed --> [*]
    dead_letter --> [*]
    cancelled --> [*]
Loading

3. Claim path — why no two workers get the same job

sequenceDiagram
    autonumber
    participant W1 as worker-1
    participant W2 as worker-2
    participant PG as PostgreSQL

    Note over W1,W2: both poll the same queue at the same instant

    W1->>PG: BEGIN — lock queues row FOR UPDATE
    W2->>PG: BEGIN — waits on the queue row
    W1->>PG: SELECT ... WHERE status='queued'<br/>FOR UPDATE SKIP LOCKED LIMIT n
    PG-->>W1: rows 1..n (now locked)
    W1->>PG: UPDATE → claimed, lease = now()+30s
    W1->>PG: COMMIT
    PG-->>W2: queue row released
    W2->>PG: SELECT ... FOR UPDATE SKIP LOCKED
    Note right of PG: rows 1..n are gone from<br/>the partial index — skipped, not blocked
    PG-->>W2: rows n+1..2n
    W2->>PG: COMMIT

    Note over W1,W2: zero overlap — verified 1000 jobs × 10 workers
Loading

4. Crash recovery

sequenceDiagram
    autonumber
    participant W as worker
    participant PG as PostgreSQL
    participant R as scheduler (reaper)

    W->>PG: claim → lock_expires_at = now()+30s, attempt++
    W->>PG: INSERT job_executions (execution_seq = 1)
    loop every 5s
        W->>PG: heartbeat — extend lease by 30s
    end

    Note over W: SIGKILL — no clean shutdown

    Note over PG: lease stops being extended

    loop every 10s
        R->>PG: SELECT WHERE lock_expires_at < now()<br/>AND status IN (claimed, running)
    end
    R->>PG: UPDATE → queued, worker_id = NULL,<br/><b>attempt = attempt - 1</b>
    Note right of R: crash is infrastructure failure,<br/>not job failure

    W->>PG: (new worker) claim → attempt = 1 again
    W->>PG: INSERT job_executions (execution_seq = 2)
    Note over PG: seq 1 still 'running' — the audit trail<br/>records the interruption rather than erasing it
Loading