Repository navigation
Leaner storage: store only events the row can't say, 15 → 9 indexes, HOT renewals - #86
Merged
Merged
Conversation
runnerq-spec now defines each event type's detail (schema/postgres/events.schema.json), and its conformance runner checks every event against it. To match: - Enqueued and Scheduled carry activity_type and priority as a number (it was a string), Scheduled its scheduled_at; Enqueued drops a null scheduled_at. - Dequeued carries activity_type with its lease deadline. - Completed drops activity_type; ResultStored drops result_stored. - Failed drops retryable; a dead letter from a failure says reason attempts_exhausted. - Requeued carries the error. - A plain Yield records result_id null and ready false, like any park.
…HOT renewals Follows runnerq-spec lean-storage (b24225d). - Submission, claims, success and checkpoints write no event; an event ending an attempt (Retrying, Failed, DeadLetter, Requeued) records the attempt's started_at and its claim as worker_id, the reaper's included. - ExtendLease (unfenced, event-writing) is gone; renewals are ExtendLeaseForWorker, now HOT updates. - RetentionPolicy.Events / RetentionConfig.Events trims events of finished activities sooner than their tree. - Retention finds expired roots with one short walk of idx_runnerq_root_terminal per status, then locks and rechecks the candidate. - Trees are id = root OR (root_activity_id = root AND parent IS NOT NULL); key locks and deletes go through the primary key; the pin check and the dependency delete use idx_runnerq_dependencies_result. - Query filters: statuses as literals (idx_runnerq_query_status), root_id through idx_runnerq_root_children. - Schema init drops retired indexes after the concurrent builds.
alob-mtc
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows runnerq/runnerq-spec#1 (spec
b24225d): store only the events the activity row can't say, cutrunnerq_activitiesfrom 15 indexes to 9, and make lease renewals HOT. This PR began as "write stored events in the spec's shapes", which this supersedes. The remaining events keep those shapes.Measured with this SDK on Postgres 17 (5,000 activities): WAL for submit, claim and complete goes from 5,296 B to 2,545 B per activity, a lease renewal from 856 B to 182 B, and storage from 2,636 B to 1,354 B.
Events
runnerq_resultssay them.Retrying,Failed,DeadLetter,Requeued) records the attempt'sstarted_atand its claim asworker_id. The reaper's events do too; they used to have no worker.ExtendLease(unfenced, used only by a test, and it wroteLeaseExtended) is removed fromstorage.Storage; renewals areExtendLeaseForWorker.Retention
RetentionConfig.Events,storage.RetentionPolicy.Events) trims events of finished activities sooner than their tree.idx_runnerq_root_terminalper status; the candidate is then locked and rechecked. Before, every sweep sorted all the queue's finished roots.Queries rewritten to use the indexes
id = root OR (root_activity_id = root AND parent_activity_id IS NOT NULL), served byidx_runnerq_root_children.idx_runnerq_dependencies_result; the pin check used to scan every dependency in the queue.idx_runnerq_query_status(which excludes completed activities) can serve them.root_idfilters use the root index.Schema
Tested locally against Postgres 17:
go test ./...passes. Conformance: 41 passed with this driver alone (3 mixed scenarios skipped), 44 passed in the mixed Go/TS fleet.Companion: runnerq/runnerq-ts#27.