feat: guarantee exactly-once alert effects across workers and delivery channels - #1118
Merged
Mosas2000 merged 1 commit intoAug 25, 2026
Conversation
…y channels Closes StellaBridge#1022 - Add 051_alert_effect_idempotency.ts migration: creates alert_effect_records table with unique constraint on effect_key (outboxEventId:channel), status domain constraint (pending/delivered/ambiguous/duplicate_suppressed), lease columns (claimed_by, claimed_at, lease_expires_at), and indexes for lease recovery sweeps, operator dashboards, and per-event lookups - Add AlertEffectGuard service implementing the exactly-once protocol: claimEffect() uses INSERT ... ON CONFLICT (effect_key) DO NOTHING for atomic single-winner claim across concurrent workers; commitEffect() guards delivery commit behind claimed_by ownership check so stolen leases cannot be committed; markAmbiguous() records permanently unclear delivery outcomes for operator resolution without triggering automatic retries; reclaimExpiredLeases() resets stuck pending claims after lease_expires_at to recover from worker crashes at any transaction boundary; recordDuplicateSuppression() appends an audit row when a manual replay or retry is blocked by an existing record - getEffectMetrics() exposes pending, delivered, ambiguous, and duplicate_suppressed counts with per-channel breakdown for Prometheus scraping and operator dashboards - buildEffectKey() produces a deterministic, stable ${outboxEventId}:${channel} key across retries, process restarts, and multiple workers - 50 unit tests covering all 5 acceptance criteria: single effect per channel per transition, crash-boundary convergence, replay duplicate suppression, ambiguous visibility for operator resolution, and metrics completeness
Contributor
|
This elegantly bridges the gap between REST state and WebSocket streams, eliminating race conditions with robust catch-up logic and stale cache detection, thank you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1022
Summary
051_alert_effect_idempotency.tsmigration that creates thealert_effect_recordstable — the idempotency ledger for exactly-once delivery. The unique constraint oneffect_key(outboxEventId:channel) is the core guard;INSERT ... ON CONFLICT DO NOTHINGmakes claim atomicity a property of the database, not the application layerAlertEffectGuardservice implementing the full exactly-once protocol:claimEffect(eventId, channel, workerId, leaseMs)— atomic single-winner claim across concurrent workers; returns{ claimed, isDuplicate, record }so callers know immediately whether to deliver or skipcommitEffect(eventId, channel, workerId)— guards the delivered transition behindclaimed_by = workerIdso a worker whose lease was stolen by crash-recovery cannot commit over a new owner's claimmarkAmbiguous(eventId, channel, workerId, reason)— records permanently unclear delivery outcomes (e.g. webhook 5xx after network timeout) without triggering automatic retries; records remain visible for operator resolutionreclaimExpiredLeases(newWorkerId)— resets expiredpendingclaims so worker crashes at any transaction boundary converge to the same final state rather than leaving slots stuck foreverrecordDuplicateSuppression(eventId, channel, attemptedBy, reason)— appends an audit row with a unique key when a manual replay or retry is blocked, preserving full audit history without touching the original recordgetEffectMetrics()— counts pending, delivered, ambiguous, and duplicate-suppressed with per-channel breakdown for Prometheus scraping and operator dashboardsbuildEffectKey(eventId, channel)— deterministic${eventId}:${channel}key stable across retries, process restarts, and multiple workersTest plan