Repository navigation
Leaner storage: store only events the row can't say, 15 → 9 indexes, HOT renewals - #27
Merged
Merged
Conversation
runnerq-spec now defines each event type's detail (schema/postgres/events.schema.json), and its conformance runner checks every event against it. To match: - Scheduled carries its scheduled_at. - Dequeued carries the lease deadline. - Retrying carries retry_count and scheduled_at instead of retryable; Failed drops retryable; a dead letter from a failure says reason attempts_exhausted instead of retryable. - Requeued carries retry_count.
…HOT renewals Follows runnerq-spec lean-storage (b24225d), as runnerq-go does. - Submission, claims, success and checkpoints write no event; an event ending an attempt records the attempt's started_at and its claim as worker_id, the reaper's included. - Retention.eventsMs trims events of finished activities sooner than their tree. - Retention finds expired roots with one short walk of idx_runnerq_root_terminal per status, then locks and rechecks the candidate. - Trees are id = root OR (root_activity_id = root AND parent IS NOT NULL); key locks and deletes go through the primary key; the pin check and the dependency delete use idx_runnerq_dependencies_result. - Query filters: statuses as literals (idx_runnerq_query_status), root_id through idx_runnerq_root_children. - initialize drops retired indexes after the concurrent builds; connect accepts a database without the new indexes.
alob-mtc
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows runnerq/runnerq-spec#1 (spec
b24225d): store only the events the activity row can't say, cutrunnerq_activitiesfrom 15 indexes to 9, and make lease renewals HOT. This PR began as "write stored events in the spec's shapes", which this supersedes. The remaining events keep those shapes.Measured with this SDK on Postgres 17 (5,000 activities): WAL for submit, claim and complete goes from 5,296 B to 2,545 B per activity, a lease renewal from 856 B to 182 B, and storage from 2,636 B to 1,354 B.
Events
runnerq_resultssay them.Retrying,Failed,DeadLetter,Requeued) records the attempt'sstarted_atand its claim asworker_id. The reaper's events do too; they used to have no worker.Retention
retention.eventsMs) trims events of finished activities sooner than their tree.idx_runnerq_root_terminalper status; the candidate is then locked and rechecked. Before, every sweep sorted all the queue's finished roots.Queries rewritten to use the indexes
id = root OR (root_activity_id = root AND parent_activity_id IS NOT NULL), served byidx_runnerq_root_children.idx_runnerq_dependencies_result; the pin check used to scan every dependency in the queue.idx_runnerq_query_status(which excludes completed activities) can serve them.root_idfilters use the root index.Schema
connectaccepts a database that lacks the new indexes or still has retired ones;initializecompletes it.Tested locally against Postgres 17: unit and integration suites pass (126/126 integration). Conformance: 41 passed with this driver alone (3 mixed scenarios skipped), 44 passed in the mixed Go/TS fleet.
Companion: runnerq/runnerq-go#86.