feat(pg-pg): introduce new MERGE statement leveraging jsonb_to_record by Amogh-Bharadwaj · Pull Request #4167 · PeerDB-io/peerdb

Amogh-Bharadwaj · 2026-04-13T06:50:49Z

Note

This feature is only for mirrors with the PG type system

Why

For workloads on source involving very wide tables and data, normalize times for PG to PG mirrors were seen to be considerably high, causing data staleness on target.

Research showed that the ->> JSON access we do for the _peerdb_data column in our raw table in Postgres is slow.

This PR is an optimization we found where instead of having to access every field in _peerdb_data, we can use jsonb_to_record to first convert the entire JSON value to a row tuple and access its fields, which is much faster.

For a 170GB table with 158 columns on Postgres receiving updates, this new MERGE statement was seen to be ~10x faster.

What

In client.go, declares a template for a new MERGE statement using jsonb_to_record.
Adapts normalize.go to use this new MERGE
Adapts unit tests
Adds type test in E2E for PG type system

codecov · 2026-04-13T07:07:03Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2210	2	2208	201

View the top 3 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg

Stack Traces | 0.01s run time

=== RUN   TestApiPg
=== PAUSE TestApiPg
=== CONT  TestApiPg
--- FAIL: TestApiPg (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH_Cluster

Stack Traces | 0.01s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH_Cluster (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg/TestCancelTableAddition_NoRemovalAssumed

Stack Traces | 14.2s run time

=== RUN   TestApiPg/TestCancelTableAddition_NoRemovalAssumed
=== PAUSE TestApiPg/TestCancelTableAddition_NoRemovalAssumed
=== CONT  TestApiPg/TestCancelTableAddition_NoRemovalAssumed
2026/04/15 08:29:27 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/04/15 08:29:27 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/04/15 08:29:28 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,c2,t FROM e2e_test_bq_zu31oncm_20260415082727.\"test_softdel_iad\" ORDER BY id"
2026/04/15 08:29:28 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,c2,t FROM e2e_test_bq_zu31oncm_20260415082727.\"test_softdel_iad\" ORDER BY id"
2026/04/15 08:29:28 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_5483986575390271878 CURSOR FOR SELECT id,c1,c2,t FROM e2e_test_bq_zu31oncm_20260415082727.\"test_softdel_iad\" ORDER BY id" args=[]
2026/04/15 08:29:28 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,c2,t FROM e2e_test_bq_zu31oncm_20260415082727.\"test_softdel_iad\" ORDER BY id" channelLen=0
2026/04/15 08:29:28 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5483986575390271878
2026/04/15 08:29:28 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5483986575390271878 records=1 bytes=9849 channelLen=0
2026/04/15 08:29:28 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,c2,t FROM e2e_test_bq_zu31oncm_20260415082727.\"test_softdel_iad\" ORDER BY id" rows=1 bytes=9849 channelLen=0
2026/04/15 08:29:28 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5483986575390271878
2026/04/15 08:29:28 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5483986575390271878 records=0 bytes=0 channelLen=0
2026/04/15 08:29:28 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,c2,t FROM e2e_test_bq_zu31oncm_20260415082727.\"test_softdel_iad\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/15 08:29:28 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/15 08:29:28 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,c2,t FROM e2e_test_bq_zu31oncm_20260415082727.\"test_softdel_iad\" ORDER BY id" rows=1 bytes=9849 channelLen=0
    cancel_table_addition_test.go:362: WaitFor wait for initial load to finish 2026-04-15 08:29:35.731166093 +0000 UTC m=+561.020123335
    cancel_table_addition_test.go:366: WaitFor t1 initial load 2026-04-15 08:29:35.73176631 +0000 UTC m=+561.020723542
2026/04/15 08:29:35 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t1\" ORDER BY id"
2026/04/15 08:29:35 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t1\" ORDER BY id"
2026/04/15 08:29:35 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_9088133834065854355 CURSOR FOR SELECT id,val FROM e2e_test_api_hi35aany.\"t1\" ORDER BY id" args=[]
2026/04/15 08:29:35 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t1\" ORDER BY id" channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_9088133834065854355
2026/04/15 08:29:35 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_9088133834065854355 records=1 bytes=9 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t1\" ORDER BY id" rows=1 bytes=9 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_9088133834065854355
2026/04/15 08:29:35 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_9088133834065854355 records=0 bytes=0 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t1\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/15 08:29:35 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t1\" ORDER BY id" rows=1 bytes=9 channelLen=0
    cancel_table_addition_test.go:367: WaitFor t2 initial load 2026-04-15 08:29:35.745640979 +0000 UTC m=+561.034598221
2026/04/15 08:29:35 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t2\" ORDER BY id"
2026/04/15 08:29:35 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t2\" ORDER BY id"
2026/04/15 08:29:35 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_11944477468628917301 CURSOR FOR SELECT id,val FROM e2e_test_api_hi35aany.\"t2\" ORDER BY id" args=[]
2026/04/15 08:29:35 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t2\" ORDER BY id" channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11944477468628917301
2026/04/15 08:29:35 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11944477468628917301 records=1 bytes=9 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t2\" ORDER BY id" rows=1 bytes=9 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11944477468628917301
2026/04/15 08:29:35 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11944477468628917301 records=0 bytes=0 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t2\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/15 08:29:35 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t2\" ORDER BY id" rows=1 bytes=9 channelLen=0
    cancel_table_addition_test.go:368: WaitFor t3 initial load 2026-04-15 08:29:35.750983703 +0000 UTC m=+561.039940944
2026/04/15 08:29:35 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t3\" ORDER BY id"
2026/04/15 08:29:35 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t3\" ORDER BY id"
2026/04/15 08:29:35 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_18414992801584121660 CURSOR FOR SELECT id,val FROM e2e_test_api_hi35aany.\"t3\" ORDER BY id" args=[]
2026/04/15 08:29:35 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t3\" ORDER BY id" channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_18414992801584121660
2026/04/15 08:29:35 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_18414992801584121660 records=1 bytes=9 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t3\" ORDER BY id" rows=1 bytes=9 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_18414992801584121660
2026/04/15 08:29:35 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_18414992801584121660 records=0 bytes=0 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t3\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/15 08:29:35 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/15 08:29:35 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_hi35aany.\"t3\" ORDER BY id" rows=1 bytes=9 channelLen=0
    cancel_table_addition_test.go:376: WaitFor wait for pause for add table 2026-04-15 08:29:35.758394011 +0000 UTC m=+561.047351253
    cancel_table_addition_test.go:377: UNEXPECTED ERROR unable to establish connection with catalog: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)
    api_test.go:48: begin tearing down postgres schema api_hi35aany
--- FAIL: TestApiPg/TestCancelTableAddition_NoRemovalAssumed (14.19s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Schema_Changes

Stack Traces | 30.5s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Schema_Changes
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Schema_Changes
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Schema_Changes
2026/04/15 08:46:42 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
    clickhouse_mysql_test.go:780: WaitFor normalize reinsert 2026-04-15 08:46:44.359563182 +0000 UTC m=+527.783036621
2026/04/15 08:46:44 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_f2xomeel.test_mysql_schema_changes
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
2026/04/15 08:46:49 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
2026/04/15 08:46:50 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
2026/04/15 08:46:51 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
2026/04/15 08:46:52 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
2026/04/15 08:46:53 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
2026/04/15 08:46:53 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
2026/04/15 08:46:54 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
2026/04/15 08:46:54 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
2026/04/15 08:46:55 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
2026/04/15 08:46:55 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
2026/04/15 08:46:57 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
    clickhouse_mysql_test.go:780: q.NumRecords: 1
    clickhouse_mysql_test.go:780: other.NumRecords: 0
2026/04/15 08:46:59 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
2026/04/15 08:46:59 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
    clickhouse_mysql_test.go:816: WaitFor normalize altered row 2026-04-15 08:47:00.597731727 +0000 UTC m=+544.021205166
2026/04/15 08:47:00 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
2026/04/15 08:47:00 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
2026/04/15 08:47:00 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_f2xomeel.test_mysql_schema_changes
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
2026/04/15 08:47:01 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
2026/04/15 08:47:02 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_9xdpx2wh.test_float
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
2026/04/15 08:47:08 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
2026/04/15 08:47:09 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
2026/04/15 08:47:10 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
    clickhouse_mysql_test.go:816: code: 47, message: Unknown expression or function identifier `addedColumn` in scope SELECT id, c1, coalesce(addedColumn, 0) AS addedColumn FROM test_mysql_schema_changes_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false. Maybe you meant: ['addedColumn']
2026/04/15 08:47:11 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
    clickhouse.go:94: 
        	Error Trace:	.../flow/e2e/congen.go:43
        	            				.../flow/e2e/clickhouse.go:94
        	            				.../flow/e2e/clickhouse.go:172
        	            				.../flow/e2e/test_utils.go:181
        	            				.../flow/e2e/test_utils.go:852
        	            				.../flow/e2e/test_utils.go:172
        	            				.../flow/e2e/clickhouse_mysql_test.go:816
        	Error:      	Received unexpected error:
        	            	unable to establish connection with catalog: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)
        	Test:       	TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Schema_Changes
2026/04/15 08:47:12 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_a84na68s.test_nullengine
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_Schema_Changes (30.48s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-04-13T07:09:50Z

❌ Test Failure

Analysis: Real bug: TestPeerFlowE2ETestSuitePG/Test_Types_PG_Legacy_Merge fails deterministically across all 3 CI jobs with a timezone mismatch (expected "06:25:00+00" but got "09:25:00+03"), indicating the pg-pg-better-merge changes broke timetz normalization in the legacy merge path.
Confidence: 0.92

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-13T07:12:00Z

❌ Test Failure

Analysis: Deterministic timezone mismatch in TestPeerFlowE2ETestSuitePG/Test_Types_PG_New_Merge — test expects 06:25:00+00 (UTC) but received 09:25:00+03 (UTC+3), indicating a real bug in time-with-timezone column handling during PG→PG replication.
Confidence: 0.85

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-13T12:58:12Z

❌ Test Failure

Analysis: Test_Types_QValue_New_Merge fails deterministically across all CI matrix jobs with identical data mismatches after normalization completes, indicating a real bug in type conversion rather than a flaky infrastructure issue.
Confidence: 0.9

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-13T13:40:52Z

🔄 Possible Flaky Test

Analysis: The failures are in MySQL GTID and MariaDB e2e suites (binary encoding mismatch and long-running tests) while the triggering commit only touches Postgres slot activity logic, strongly suggesting pre-existing flaky e2e tests unrelated to the change.
Confidence: 0.65

⚠️ Confidence too low (0.65) to retry automatically - manual review recommended

View workflow run

github-actions · 2026-04-13T13:47:14Z

❌ Test Failure

Analysis: Tests Test_Types_PG_New_Merge and Test_Types_PG_Legacy_Merge fail with a real timezone normalization bug — the merge produces 09:25:00+03 instead of the expected 06:25:00+00 for timetz values, a consistent data correctness regression introduced by this PR's changes to PG-to-PG merge logic.
Confidence: 0.92

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-13T13:51:59Z

❌ Test Failure

Analysis: Real bug: the pg-pg-better-merge PR introduced a regression in timetz type handling — Test_Types_PG_New_Merge, Test_Types_PG_Legacy_Merge, and Test_Types_QValue_New_Merge all fail with a deterministic assertion error (expected: "06:25:00+00") across all three CI matrix configurations.
Confidence: 0.88

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-13T13:57:10Z

❌ Test Failure

Analysis: Real build failure: the field useJsonbToRecord was removed from normalizeStmtGenerator in the recent normalize race fix (b0735da), but normalize_stmt_generator_test.go still references it in 9 struct literals, causing a compilation error.
Confidence: 0.98

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-13T14:04:14Z

❌ Test Failure

Analysis: Test_Types_QValue fails deterministically across all 3 CI matrix configurations with a data mismatch in normalized column values, likely a regression introduced by the recent "fix normalize race" commit touching the Postgres target normalize path.
Confidence: 0.8

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-13T14:18:19Z

❌ Test Failure

Analysis: Test_Types_QValue fails consistently at ~197s across all matrix variants, likely a regression from the recent postgres normalization fix rather than a flaky timeout.
Confidence: 0.78

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-13T15:11:35Z

❌ Test Failure

Analysis: TestPeerFlowE2ETestSuitePG/Test_Types_QValue fails consistently across all 3 CI matrix variants at ~197s with the same row mismatch, strongly suggesting a real regression in PG-to-PG normalization likely introduced by the ORDER BY in MERGE change in the latest commit.
Confidence: 0.72

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-15T08:35:23Z

🔄 Possible Flaky Test

Analysis: Unable to retrieve failure logs as the run is still in progress; low-confidence flaky guess based on the pattern of two sibling matrix jobs (pg16/mysql-gtid, pg18/maria) passing while only the pg17/mysql-pos job is still running.
Confidence: 0.4

⚠️ Confidence too low (0.4) to retry automatically - manual review recommended

View workflow run

github-actions · 2026-04-15T08:52:49Z

🔄 Flaky Test Detected

Analysis: The e2e test suite failed only in the mysql-pos matrix configuration (768s runtime, near the 900s limit) while identical code passed in maria and mysql-gtid configurations, indicating a timing/infrastructure flake rather than a code regression.
Confidence: 0.85

✅ Automatically retrying the workflow

View workflow run

Amogh-Bharadwaj requested review from heavycrystal and serprex April 13, 2026 06:50

serprex reviewed Apr 13, 2026

View reviewed changes

Comment thread flow/internal/dynamicconf.go Outdated

Amogh-Bharadwaj changed the title ~~feat(pg-pg): introduce code path for jsonb record in merge behind feature flag~~ feat(pg-pg): improve MERGE statement by leveraging jsonb_to_record Apr 13, 2026

serprex reviewed Apr 13, 2026

View reviewed changes

Comment thread flow/e2e/postgres_test.go Outdated

serprex approved these changes Apr 13, 2026

View reviewed changes

Amogh-Bharadwaj force-pushed the pg-pg-better-merge branch from bc8f275 to c11670f Compare April 13, 2026 14:50

introduce jsonb to record

dfb8796

Amogh-Bharadwaj force-pushed the pg-pg-better-merge branch from c11670f to dfb8796 Compare April 15, 2026 08:16

Amogh-Bharadwaj changed the title ~~feat(pg-pg): improve MERGE statement by leveraging jsonb_to_record~~ feat(pg-pg): introduce new MERGE statement leveraging jsonb_to_record Apr 15, 2026

Amogh-Bharadwaj merged commit 7d8be24 into main Apr 15, 2026
18 of 20 checks passed

Amogh-Bharadwaj deleted the pg-pg-better-merge branch April 15, 2026 09:13

Conversation

Amogh-Bharadwaj commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Uh oh!

codecov bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

github-actions bot commented Apr 13, 2026

❌ Test Failure

Uh oh!

github-actions bot commented Apr 13, 2026

❌ Test Failure

Uh oh!

github-actions bot commented Apr 13, 2026

❌ Test Failure

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 13, 2026

🔄 Possible Flaky Test

Uh oh!

github-actions bot commented Apr 13, 2026

❌ Test Failure

Uh oh!

github-actions bot commented Apr 13, 2026

❌ Test Failure

Uh oh!

github-actions bot commented Apr 13, 2026

❌ Test Failure

Uh oh!

github-actions bot commented Apr 13, 2026

❌ Test Failure

Uh oh!

github-actions bot commented Apr 13, 2026

❌ Test Failure

Uh oh!

github-actions bot commented Apr 13, 2026

❌ Test Failure

Uh oh!

github-actions bot commented Apr 15, 2026

🔄 Possible Flaky Test

Uh oh!

github-actions bot commented Apr 15, 2026

🔄 Flaky Test Detected

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Amogh-Bharadwaj commented Apr 13, 2026 •

edited

Loading

codecov bot commented Apr 13, 2026 •

edited

Loading