Skip to content

feat: guarantee exactly-once alert effects across workers and delivery channels - #1118

Merged
Mosas2000 merged 1 commit into
StellaBridge:mainfrom
stableprogrammer:feat/exactly-once-alert-effects-1022
Aug 25, 2026
Merged

feat: guarantee exactly-once alert effects across workers and delivery channels#1118
Mosas2000 merged 1 commit into
StellaBridge:mainfrom
stableprogrammer:feat/exactly-once-alert-effects-1022

Conversation

@stableprogrammer

Copy link
Copy Markdown
Contributor

Closes #1022

Summary

  • Adds 051_alert_effect_idempotency.ts migration that creates the alert_effect_records table — the idempotency ledger for exactly-once delivery. The unique constraint on effect_key (outboxEventId:channel) is the core guard; INSERT ... ON CONFLICT DO NOTHING makes claim atomicity a property of the database, not the application layer
  • Adds AlertEffectGuard service implementing the full exactly-once protocol:
    • claimEffect(eventId, channel, workerId, leaseMs) — atomic single-winner claim across concurrent workers; returns { claimed, isDuplicate, record } so callers know immediately whether to deliver or skip
    • commitEffect(eventId, channel, workerId) — guards the delivered transition behind claimed_by = workerId so a worker whose lease was stolen by crash-recovery cannot commit over a new owner's claim
    • markAmbiguous(eventId, channel, workerId, reason) — records permanently unclear delivery outcomes (e.g. webhook 5xx after network timeout) without triggering automatic retries; records remain visible for operator resolution
    • reclaimExpiredLeases(newWorkerId) — resets expired pending claims so worker crashes at any transaction boundary converge to the same final state rather than leaving slots stuck forever
    • recordDuplicateSuppression(eventId, channel, attemptedBy, reason) — appends an audit row with a unique key when a manual replay or retry is blocked, preserving full audit history without touching the original record
    • getEffectMetrics() — counts pending, delivered, ambiguous, and duplicate-suppressed with per-channel breakdown for Prometheus scraping and operator dashboards
    • buildEffectKey(eventId, channel) — deterministic ${eventId}:${channel} key stable across retries, process restarts, and multiple workers

Test plan

  • AC-1: first claim returns claimed=true; second claim on same effect_key returns claimed=false, isDuplicate=true
  • AC-1: two different channels on one event can both be claimed independently
  • AC-1: concurrent claims produce exactly one winner
  • AC-2: commit by owning worker marks delivered; commit by wrong worker returns false
  • AC-2: crash after claim, before commit: reclaimExpiredLeases resets the slot; recovery worker commits; exactly one delivered record results
  • AC-2: crash after commit: retry attempt finds isDuplicate=true, no second delivery
  • AC-3: replay after delivered effect is blocked (isDuplicate=true) and recorded as duplicate_suppressed
  • AC-4: markAmbiguous records the reason; listAmbiguous returns the record; reclaimExpiredLeases does not touch ambiguous records
  • AC-5: getEffectMetrics counts all four statuses; byChannel breakdown matches per-channel state
  • AC-5: total equals sum of all per-status counts

…y channels

Closes StellaBridge#1022

- Add 051_alert_effect_idempotency.ts migration: creates alert_effect_records
  table with unique constraint on effect_key (outboxEventId:channel), status
  domain constraint (pending/delivered/ambiguous/duplicate_suppressed), lease
  columns (claimed_by, claimed_at, lease_expires_at), and indexes for lease
  recovery sweeps, operator dashboards, and per-event lookups
- Add AlertEffectGuard service implementing the exactly-once protocol:
  claimEffect() uses INSERT ... ON CONFLICT (effect_key) DO NOTHING for
  atomic single-winner claim across concurrent workers; commitEffect() guards
  delivery commit behind claimed_by ownership check so stolen leases cannot
  be committed; markAmbiguous() records permanently unclear delivery outcomes
  for operator resolution without triggering automatic retries; reclaimExpiredLeases()
  resets stuck pending claims after lease_expires_at to recover from worker
  crashes at any transaction boundary; recordDuplicateSuppression() appends
  an audit row when a manual replay or retry is blocked by an existing record
- getEffectMetrics() exposes pending, delivered, ambiguous, and duplicate_suppressed
  counts with per-channel breakdown for Prometheus scraping and operator dashboards
- buildEffectKey() produces a deterministic, stable ${outboxEventId}:${channel}
  key across retries, process restarts, and multiple workers
- 50 unit tests covering all 5 acceptance criteria: single effect per channel
  per transition, crash-boundary convergence, replay duplicate suppression,
  ambiguous visibility for operator resolution, and metrics completeness
@Mosas2000

Copy link
Copy Markdown
Contributor

This elegantly bridges the gap between REST state and WebSocket streams, eliminating race conditions with robust catch-up logic and stale cache detection, thank you.

@Mosas2000
Mosas2000 merged commit 72d953f into StellaBridge:main Aug 25, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Guarantee Exactly-Once Alert Effects Across Workers and Delivery Channels

2 participants