feat: repo materialization worker (disabled for now)#5555
Open
mchalapuk wants to merge 17 commits into
Open
Conversation
Adds the git-backed canvas materialization engine (pkg/canvas/materialize) and the RepositoryMaterializer worker that consumes repository-branch-updated messages and projects git branch tips into the database. Wires the worker into the server behind START_REPOSITORY_MATERIALIZER and bumps supergit to 0.1.2 in the dev compose for branch/merge support. The worker is disabled for now: START_REPOSITORY_MATERIALIZER is "no" in the dev compose and is left unset in the release/prod configs (the server only starts it when the flag is "yes"), so it can be enabled later without code changes. All materialization state (commit SHA, branch, status, error) is tracked on the existing workflow_versions table, so no additional table is required. Model status strings are mapped to the typed MaterializationStatus proto enum at the message boundary. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
|
👋 Commands for maintainers:
|
superplane-policy-bot
Bot
requested review from
forestileao,
lucaspin and
shiroyasha
Move git provider reads (ListBranches, Head, LoadRepoSnapshot) out of the database transactions in the materialization engine so no git RPC is held across a pooled DB connection. The sync functions now load git state first and open their own short transaction for DB writes only; MaterializeLive and MaterializeDraft take a pre-loaded snapshot. The live publisher's node Setup() work stays inside the transaction by design (atomic with the node rows). Also wire fully async draft-branch deletion: the worker now reconciles the DB projection when it receives a DELETED notification. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
mchalapuk
commented
Jun 17, 2026
Move the canvas spec file names and draft branch naming/lookup helpers (IsDraftBranch, DefaultDraftBranchName, OwnerFromDraftBranchName, UniqueDraftBranchName, GitBranchExists) out of the materialize engine into a new leaf package pkg/canvas/gitref. The engine and the git-repo seeding/backfill code can both depend on it without a cycle. Display-name helpers stay in the engine since only it uses them. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
Move seed_repository.go, seed_yaml.go and backfill.go out of the materialization engine into a new pkg/canvas/gitrepo package, which owns building the git representation of a canvas (YAML encoders, initial repo seeding, and pre-git-first backfill). Rename the YAML builders to the *ToBytes convention (CanvasYAMLToBytes, ConsoleYAMLToBytes, EmptyConsoleYAMLToBytes, ConsoleYAMLFromVersionToBytes) so callers read clearly as "encode to bytes". Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
Move RequestBranchMaterialization and the test-only in-process materializer toggle (SetInProcessMaterializer/inProcessMaterializer) out of the materialize engine and into pkg/workers, next to the consumer that runs the work. The engine keeps only the BranchMaterializer core (renamed file to branch_materializer.go). The producer publishes the pending status enum directly, so no status mapping needs to be exported. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
…-only Delete the unused Materializer/MaterializeFromGit/Mode indirection; the BranchMaterializer is the single entry point. MaterializeDraft now only handles draft branches (it is only ever called for drafts): drop the dead non-draft commit-SHA lookup branch and the always-true draft guards, and reject non-draft branches defensively. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
…n head SyncLiveFromGit now always reads the current main HEAD and, when given an explicit notification SHA that no longer matches it, skips as a no-op instead of projecting a superseded commit. The newer commit that is now main's HEAD carries its own notification and materializes current state. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
Trim the engine's public surface to BranchMaterializer (the only type the worker needs). Unexport the sync orchestrators, live/draft writers, snapshot loader and type, reconcile helpers, and their options structs. The snapshot-loader test becomes an internal package test so it can still exercise the loader directly. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
Add unit tests for the git-first materialization engine: live/draft branch writers, stale-HEAD live skip, draft-deletion reconciliation, and the canvas repository backfill (main + draft spec files). Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
mchalapuk
force-pushed
the
feat/git-materialization-worker
branch
from
3de8310 to
70f171a
Compare
Switch the draft-deletion reconciler and the repository backfill to read git_branch (the canonical branch field) instead of branch_name. Writers still populate branch_name because the draft CHECK constraint and the model lookup/upsert helpers depend on it until the column-drop migration lands in a separate PR. Emulate that migration's git_branch backfill in the backfill test fixture. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the owner-encoded draft branch scheme (DefaultDraftBranchName / OwnerFromDraftBranchName / UniqueDraftBranchName) with a single NewDraftBranchName helper that returns drafts/<random-uuid>. Draft branches no longer encode ownership; the materializer now records the pusher as the draft owner. This also matches how the model already names draft branches and fixes OwnerFromDraftBranchName misreading the random branch uuid as a user id. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
mchalapuk
changed the title
Split the materialization engine into per-branch files (live_materializer, draft_materializer, draft_deleter) that own their own transactions and idempotency checks, route error/deletion signals through canvas_version_updated, gate the worker to actionable requests, and fold the gitref vocabulary (spec file names, draft branch prefix, IsDraftBranch) into the models package. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
Drop seed/backfill helpers that have no production callers yet and inline minimal git fixture setup in materialize tests instead. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
mchalapuk
force-pushed
the
feat/git-materialization-worker
branch
from
4104cd7 to
c68ecec
Compare
Draft version rows now rely on git_branch only when materializing drafts and persisting materialization errors. Signed-off-by: Maciej Chałapuk <maciej@chalapuk.pl> Co-authored-by: Cursor <cursoragent@cursor.com>
mchalapuk
changed the title
PR Risk ReviewRisk: 42/100 (medium) SummaryNew repository materialization worker feature that is explicitly disabled (START_REPOSITORY_MATERIALIZER=no) with proper advisory locking, idempotency guards, comprehensive tests, and a clean separation of concerns. Concerns
Recommended reviewers: superplanehq/backend-team |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the git-backed canvas materialization engine (
pkg/canvas/materialize) and theRepositoryMaterializerworker that consumesrepository-branch-updatedmessages and projects git branch tips intoworkflow_versions.Details
START_REPOSITORY_MATERIALIZERand bumps supergit to0.1.2(branch/merge support)."no"in the dev compose and unset in prod/release configs, so it can be enabled later without code changes.workflow_versionstable.