Skip to content

feat: share one npm install across work directories - #56

Open
donbowman wants to merge 2 commits into
wey-gu:mainfrom
donbowman:feat/shared-node-modules
Open

donbowman wants to merge 2 commits into
wey-gu:mainfrom
donbowman:feat/shared-node-modules

Conversation

@donbowman

Copy link
Copy Markdown

Summary

Parallel test workers (for example pytest-xdist with -n 8) each install the PGlite npm packages into their own work directory today, so a test run does one npm install per worker. This adds a node_modules_dir config option that installs them once, guarded by a cross-process file lock, and links the result into each work directory:

config = PGliteConfig(
    work_dir=Path(f"/tmp/pglite-worker-{worker_id}"),
    node_modules_dir=Path("/tmp/py-pglite-npm"),
)
  • The first worker installs; the others wait on the lock and reuse it.
  • The lock (flock / msvcrt.locking) is released by the OS if the installer dies, so a crashed process cannot leave a stale lock.
  • Changing the dependency set (enabling an extension, upgrading py-pglite) invalidates and reinstalls the shared copy automatically.
  • Behaviour is unchanged when node_modules_dir is not set (install into the work directory).

Package alignment

The branch moves the npm pins to the latest compatible releases:

  • @electric-sql/pglite ^0.5.8
  • @electric-sql/pglite-socket ^0.2.11
  • @electric-sql/pglite-pgvector ^0.0.9

Since pglite 0.5.x moved vector out of the main bundle, SUPPORTED_EXTENSIONS["pgvector"] now points at @electric-sql/pglite-pgvector, matching #52's approach.

Readiness fix

With the newer socket server, the existing readiness probe (bare connect + immediate close) causes the next connection to be dropped. Two changes make startup deterministic:

  • the generated script awaits db.waitReady before server.start(), so the socket only accepts once PGlite can serve queries;
  • the Python readiness check performs a real PostgreSQL startup handshake (StartupMessage, AuthenticationOk / cleartext password, ReadyForQuery, graceful Terminate) instead of a bare socket probe.

Without this, tests/test_extensions.py::test_pgvector_extension fails with server closed the connection unexpectedly on the new packages.

Overlap

Testing

  • tests/test_shared_node_modules.py: shared install reuse, stale-install invalidation, linking over an existing directory, lock timeout, package pins, startup probe, generated-script ordering.
  • tests/test_extensions.py::test_pgvector_extension passes with the new packages.
  • Core manager / advanced / connection-management slices: 89 passed locally.

Add a node_modules_dir config option that installs the PGlite npm
packages once and links them into each work directory, so parallel test
workers (for example pytest-xdist) no longer race to install the same
packages in their own directories.  Installs are serialised with a file
lock that the OS releases if the installer dies, and the shared install
is refreshed automatically when the dependencies change.

The npm pins move to the latest compatible releases (pglite 0.5.8,
pglite-socket 0.2.11, pglite-pgvector 0.0.9), which means pgvector now
comes from its own package, and the generated server script waits for
db.waitReady before accepting connections.  The readiness probe now
performs a real PostgreSQL startup handshake instead of a bare
connect-and-close, because the newer socket server drops the next
connection after an abrupt probe disconnect.
The readiness probe performs a PostgreSQL startup handshake, so tests
that mock socket.socket and call start() must stub
_postgres_startup_probe as well: a mocked socket cannot speak the
protocol.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant