Conversation
tsg
commented
Mar 12, 2026
| } | ||
| return w.dmlAdapter.buildBulkInsertQueries(events, run[0].schemaInfo), nil | ||
| default: | ||
| // UPDATE, TRUNCATE, and anything else: individual queries |
Member
Author
There was a problem hiding this comment.
This means we can still have the same type of issue with UPDATE ... WHERE ... statements that updates a lot of rows. Hopefully less common than the commit case.
kvch
reviewed
Mar 23, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the Postgres stream-mode writer to batch/coalesce WAL DML operations (similar to snapshot mode) by deferring query building until batch flush time, enabling bulk SQL generation for higher throughput during catch-up.
Changes:
- Introduce
walMessagebatching and awalEventToMessageadapter path to defer DML query construction until batch send time. - Add bulk/coalesced DML query builders for streaming (notably multi-row INSERT and coalesced DELETE).
- Add unit tests for
walMessagesizing/emptiness, bulk DML adapters, updated batch-writer behavior, plus new pg→pg integration tests for coalescing.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/wal/processor/postgres/postgres_writer.go | Adds a batch-sender interface for walMessage batching. |
| pkg/wal/processor/postgres/postgres_wal_message.go | Introduces walMessage wrapper with size estimation for batching. |
| pkg/wal/processor/postgres/postgres_wal_message_test.go | Unit tests for walMessage IsEmpty and Size. |
| pkg/wal/processor/postgres/postgres_wal_dml_adapter_bulk.go | Adds bulk INSERT and coalesced DELETE query builders used during streaming batch flush. |
| pkg/wal/processor/postgres/postgres_wal_dml_adapter_bulk_test.go | Unit tests for bulk delete/insert query builders and parameter splitting. |
| pkg/wal/processor/postgres/postgres_wal_adapter.go | Adds walEventToMessage for message-based batching. |
| pkg/wal/processor/postgres/postgres_wal_adapter_test.go | Adds tests covering walEventToMessage behavior. |
| pkg/wal/processor/postgres/postgres_batch_writer.go | Switches stream batch writer to batch walMessage and coalesce consecutive same-table DML. |
| pkg/wal/processor/postgres/postgres_batch_writer_test.go | Updates tests to match message-based batching and validates coalescing behavior. |
| pkg/wal/processor/postgres/instrumented_wal_adapter.go | Instruments walEventToMessage with OTel spans. |
| pkg/wal/processor/postgres/helper_test.go | Extends mocks to support walEventToMessage. |
| pkg/stream/integration/pg_pg_batch_coalesce_integration_test.go | New integration tests validating coalescing for inserts/deletes and composite keys. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
kvch
reviewed
Mar 23, 2026
Merging this branch will increase overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
kvch
approved these changes
Apr 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is the implementation for the plan in #770
Before this PR, we were batching write operations against the target in snapshot mode, but not in stream mode. This adds batching also in stream mode, as a performance optimisation.
Related Issue(s)
Type of Change
Please select the relevant option(s):
Testing
Checklist