Skip to content

Conversation

@JohannesBin
Copy link

Summary

Move-in messages from Electric's tagged_subqueries feature don't have an lsn header because they come from direct database queries, not from the PostgreSQL replication stream. Previously, these messages were incorrectly skipped as "already seen" because the missing LSN defaulted to 0.

This fix checks for the is_move_in header and bypasses LSN filtering for move-in messages, ensuring that rows moving into a shape due to subquery condition changes are properly synced to the client.

Problem

When using shapes with subqueries (e.g., WHERE id IN (SELECT user_id FROM workspace_members WHERE workspace_id = $1)), new rows that "move in" to match the subquery weren't being synced:

  1. Electric server correctly processes the move-in and sends an INSERT message
  2. The INSERT message has headers.is_move_in: true but NO headers.lsn (since it's from a query, not replication)
  3. pglite-sync defaults missing LSN to 0
  4. The check lsn <= lastCommittedLsnForShape evaluates to 0 <= X which is true after initial sync
  5. The move-in message is skipped as "already seen"

Solution

const isMoveIn = (message.headers as Record<string, unknown>).is_move_in === true

if (!isMoveIn && lsn <= lastCommittedLsnForShape) {
  // Skip only if NOT a move-in message
  return
}

Test case

  1. Create a shape with a subquery WHERE clause (e.g., profiles where id IN (SELECT user_id FROM workspace_members WHERE workspace_id = ...))
  2. Complete initial sync
  3. Add a new row to the inner table that causes a row to "move in" to the outer shape
  4. Before fix: the moved-in row is not synced to PGlite
  5. After fix: the moved-in row is correctly synced

Move-in messages from Electric's tagged_subqueries feature don't have
an LSN header because they come from direct DB queries, not replication.
Previously these messages were incorrectly skipped as "already seen"
because the missing LSN defaulted to 0.

This checks for the is_move_in header and bypasses LSN filtering for
move-in messages, ensuring rows moving into a shape due to subquery
condition changes are properly synced.

Fixes electric-sql/electric#3769
Move-in data from tagged_subqueries can overlap with initial sync data,
causing duplicate key errors. This adds ON CONFLICT DO UPDATE handling
specifically for move-in inserts.

- Add primaryKey param to applyInsertsToTable
- Use ON CONFLICT DO UPDATE for move-in inserts
- Update changeset description
@JohannesBin
Copy link
Author

JohannesBin commented Jan 25, 2026

Added a second commit to handle duplicate key errors that can occur when move-in data overlaps with initial sync data.

When a row "moves in" to a shape, the server sends an INSERT. But if that row was already synced during initial sync, we get a duplicate key error.

For move-in inserts specifically, use ON CONFLICT DO UPDATE to upsert instead of plain INSERT. This only applies when is_move_in: true - regular inserts still use plain INSERT to catch any unexpected duplicates as errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant