Skip to content

fix(postgres-migration): ClickPipes does the migration leg, so correct the lab code - #100

Merged
sishuo-yang merged 3 commits into
mainfrom
sishuo-yang/postgres-migration-clickpipes-lab
Aug 21, 2026
Merged

sishuo-yang merged 3 commits into
mainfrom
sishuo-yang/postgres-migration-clickpipes-lab

Conversation

@sishuo-yang

Copy link
Copy Markdown
Collaborator

The playbook half of this correction is in WorkshopHouse (private). Two lab-code files here carried the same wrong model.

Background

The workshop asserted that ClickPipes cannot move Postgres to Postgres and built a two-tool architecture on it. That is false: ClickPipes has a Managed Postgres destination — Data sources → Start import, with parallel initial load plus CDC. Module 03 now creates that pipe.

terraform/variables.tf

The comment above subscriber_cidrs said:

ClickHouse Managed Postgres creates the SUBSCRIPTION, so it dials OUT to this instance.

The mover is the ClickPipes service, dialling in from its own documented static egress addresses — not the Managed Postgres instance. This is operational, not cosmetic: allowlisting the wrong party produces a pipe that never starts loading and reports no error anywhere, and the old comment sent a reader hunting for an address on the wrong console page.

It also resolves the open item that comment referenced ("narrow this to ClickHouse Cloud's documented egress addresses once confirmed"). They exist, published per region for ClickPipes. The description now says so too, instead of "the ClickHouse Cloud egress range".

Added: this leg does not work through a connection pooler. PgBouncer and RDS Proxy cannot carry logical decoding. This Terraform creates no proxy, so it does not bite here — but a production source usually has one.

sql/03_publication.sql

This was the publication step of a hand-rolled logical-replication leg the module no longer performs. Rather than delete it, its header now states which of its two uses applies to which path:

  • The CREATE is for doing the leg by hand with logical replication — which remains documented and is a reasonable choice if you cannot take a dependency on a public-beta console flow or you need every step in a script.
  • The two SELECTs are what the main line runs. The pipe creates a publication on the source from your table selection, so reading pg_publication.puballtables and pg_publication_tables verifies the scope that is actually in force, out of the source's own catalog, rather than the form you filled in. That also catches a pipe whose table selection was wider than intended, which otherwise silently bypasses the workshop's replication-scope decision.

The scope rationale is unchanged and now phrased tool-independently: an explicit list records a decision and holds it; replicating everything enrols whatever exists when the leg starts.

Verification

  • terraform fmt -check and terraform validate pass
  • both files diff clean against the private authoring copy, so the two trees stay in sync for the next publish

Sources: Introducing Postgres to Postgres ClickPipes · Migrate PostgreSQL data using Data sources

🤖 Generated with Claude Code

…t the lab code

The workshop's playbook said ClickPipes cannot move Postgres to Postgres and
built its architecture on that. It is false -- ClickPipes has a Managed Postgres
destination -- and the playbook half of the correction is in WorkshopHouse. Two
lab-code files carried the same wrong model.

terraform/variables.tf said the Managed Postgres instance creates the
SUBSCRIPTION and therefore dials out to RDS, so subscriber_cidrs should hold its
egress address. The mover is the ClickPipes service, dialling in from its own
documented static addresses. This matters operationally rather than
cosmetically: allowlisting the wrong party produces a pipe that never starts
loading and reports no error, and the old comment sent a reader looking for an
address on the wrong console page. It also resolves the open item the comment
referenced -- static egress addresses do exist, published per region.

sql/03_publication.sql was the publication step of a hand-rolled logical
replication leg that the module no longer performs. Rather than delete it, it now
states which of its two uses applies to which path: the CREATE is for anyone
doing the leg by hand with logical replication, which remains documented and is a
reasonable choice; the two SELECTs are the inspection the main line does run,
because the pipe creates a publication on the source and reading the scope out of
pg_publication_tables verifies what will actually be replicated from rather than
the form you filled in.

Verified: terraform fmt -check and terraform validate pass; the file diffs clean
against the private authoring copy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sishuo-yang
sishuo-yang merged commit 9263a08 into main Aug 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant