Repository navigation
fix(postgres-migration): ClickPipes does the migration leg, so correct the lab code - #100
Merged
Merged
Conversation
…t the lab code The workshop's playbook said ClickPipes cannot move Postgres to Postgres and built its architecture on that. It is false -- ClickPipes has a Managed Postgres destination -- and the playbook half of the correction is in WorkshopHouse. Two lab-code files carried the same wrong model. terraform/variables.tf said the Managed Postgres instance creates the SUBSCRIPTION and therefore dials out to RDS, so subscriber_cidrs should hold its egress address. The mover is the ClickPipes service, dialling in from its own documented static addresses. This matters operationally rather than cosmetically: allowlisting the wrong party produces a pipe that never starts loading and reports no error, and the old comment sent a reader looking for an address on the wrong console page. It also resolves the open item the comment referenced -- static egress addresses do exist, published per region. sql/03_publication.sql was the publication step of a hand-rolled logical replication leg that the module no longer performs. Rather than delete it, it now states which of its two uses applies to which path: the CREATE is for anyone doing the leg by hand with logical replication, which remains documented and is a reasonable choice; the two SELECTs are the inspection the main line does run, because the pipe creates a publication on the source and reading the scope out of pg_publication_tables verifies what will actually be replicated from rather than the form you filled in. Verified: terraform fmt -check and terraform validate pass; the file diffs clean against the private authoring copy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The playbook half of this correction is in WorkshopHouse (private). Two lab-code files here carried the same wrong model.
Background
The workshop asserted that ClickPipes cannot move Postgres to Postgres and built a two-tool architecture on it. That is false: ClickPipes has a Managed Postgres destination — Data sources → Start import, with parallel initial load plus CDC. Module 03 now creates that pipe.
terraform/variables.tfThe comment above
subscriber_cidrssaid:The mover is the ClickPipes service, dialling in from its own documented static egress addresses — not the Managed Postgres instance. This is operational, not cosmetic: allowlisting the wrong party produces a pipe that never starts loading and reports no error anywhere, and the old comment sent a reader hunting for an address on the wrong console page.
It also resolves the open item that comment referenced ("narrow this to ClickHouse Cloud's documented egress addresses once confirmed"). They exist, published per region for ClickPipes. The
descriptionnow says so too, instead of "the ClickHouse Cloud egress range".Added: this leg does not work through a connection pooler. PgBouncer and RDS Proxy cannot carry logical decoding. This Terraform creates no proxy, so it does not bite here — but a production source usually has one.
sql/03_publication.sqlThis was the publication step of a hand-rolled logical-replication leg the module no longer performs. Rather than delete it, its header now states which of its two uses applies to which path:
CREATEis for doing the leg by hand with logical replication — which remains documented and is a reasonable choice if you cannot take a dependency on a public-beta console flow or you need every step in a script.SELECTs are what the main line runs. The pipe creates a publication on the source from your table selection, so readingpg_publication.puballtablesandpg_publication_tablesverifies the scope that is actually in force, out of the source's own catalog, rather than the form you filled in. That also catches a pipe whose table selection was wider than intended, which otherwise silently bypasses the workshop's replication-scope decision.The scope rationale is unchanged and now phrased tool-independently: an explicit list records a decision and holds it; replicating everything enrols whatever exists when the leg starts.
Verification
terraform fmt -checkandterraform validatepassSources: Introducing Postgres to Postgres ClickPipes · Migrate PostgreSQL data using Data sources
🤖 Generated with Claude Code