Skip to content

feat(meta): add build.nativeDependencies for packages that cannot be bundled - #76

Merged
joshunrau merged 1 commit into
DouglasNeuroInformatics:mainfrom
thomasbeaudry:feat/native-dependencies
Aug 4, 2026
Merged

feat(meta): add build.nativeDependencies for packages that cannot be bundled#76
joshunrau merged 1 commit into
DouglasNeuroInformatics:mainfrom
thomasbeaudry:feat/native-dependencies

Conversation

@thomasbeaudry

Copy link
Copy Markdown
Contributor

Problem

buildProd collapses the dependency graph into a single file. That breaks any package
which loads a native artifact through a path relative to its own source, because the
JavaScript moves and the artifact does not.

prismaPlugin already solves exactly this, in three steps — locate the artifact
(prisma.ts:19-27), point the package back at it via the banner (prisma.ts:29), emit it
into the output directory (prisma.ts:31-33). The mechanism is right; it is just welded to
one package name, with no way for a consumer to declare a second.

esbuild is the case that motivated this, and it is the nastiest variant: it does not merely
fail to find its executable, it actively refuses to run. From
esbuild/lib/main.js:1812:

if ((!ESBUILD_BINARY_PATH || false) && (path.basename(__filename) !== "main.js" || path.basename(__dirname) !== "lib")) {
  throw new Error(`The esbuild JavaScript API cannot be bundled. Please mark the "esbuild" package as external...`);
}

Note the first clause: setting ESBUILD_BINARY_PATH both locates the executable and
suppresses the refusal.

The existing escape hatch does not help. build.bundle: false adds externalPlugin, which
externalizes all of node_modules — for a consumer whose runner image ships only the
output directory, that trades one broken build for another.

Solution

Generalize the Prisma pattern into a table-driven plugin behind a declarative option:

build: {
  nativeDependencies: ['esbuild']
}

The executable is emitted next to the bundle, so it travels with whatever already copies the
output directory — no extra deployment step. A consumer needing a package without a built-in
recipe supplies locate, outputName, packageName and runtimeEnvVar directly.

The downstream consumer that hit this deletes 14 lines of Dockerfile for that one line.

Two design points worth review attention

Resolution comes from the application's graph, not libnest's. prismaPlugin uses
module.createRequire(import.meta.url) (prisma.ts:7) — libnest's own location. That is
correct for Prisma, whose engines are a libnest dependency. It is wrong in general: libnest
depends on esbuild@^0.27.2, while the consumer that hit this resolves esbuild@0.23.1
through a transitive dependency, and its store holds four esbuild versions. Emitting the
0.27 executable beside bundled 0.23 JavaScript produces Cannot start service: Host version … does not match binary version … on first use, in production.

So the plugin hooks onResolve and takes the answer from the actual graph walk, then roots
locate's require at the module that imported the package. Resolution is observed, never
altered
— the handler always returns null and the package is still bundled normally.
This is the same build.resolve pattern externalPlugin uses, and esbuild passes
pluginName on that call so it cannot re-enter the plugin.

Declaring a dependency the application never imports fails the build. This is deliberate,
because the alternative failure is quiet. If the banner points at a file that was never
emitted, esbuild's generateBinPath only warns on a bad ESBUILD_BINARY_PATH
(main.js:1682-1690) and then falls through to normal resolution — so the operator sees
Cannot find module 'esbuild', which blames entirely the wrong thing. I verified that
failure mode directly before choosing to guard against it.

Prerequisite fixes included

Both were found while building this, and the first blocks it outright:

  • prismaPlugin appended its banner statement without a trailing semicolon
    (prisma.ts:29). The banner concatenates statements from every appending plugin with no
    separator, so a second appender produced a bundle that does not parse — confirmed with
    node --check; there is no newline for ASI to rescue. It works today only because
    prismaPlugin happens to be the only appender.
  • load.ts validated an esbuildOptions field that nothing reads. It was removed from
    the user-facing surface in a987e72, but the zod line survived, so a consumer could set it,
    have it validate cleanly, and have it silently ignored.

Verification

  • 221 tests pass, and the 100% coverage threshold holds.
  • New src/meta/plugins/__tests__/native-dependencies.test.ts (10 tests) covers the banner,
    subpath matching, resolve-once, the two throw paths, and that locate receives a require
    rooted at the importing module rather than at libnest.
  • New src/meta/__tests__/native-dependencies.test.ts covers recipe expansion and that the
    esbuild recipe asks for the current host's platform package.
  • src/meta/__tests__/build.test.ts gains three cases, including a real bundle of the
    example app that declares a native dependency and asserts both that the artifact lands
    beside the output and that the banner assignment is present in the emitted file.
  • The underlying mechanism was verified independently before implementation: a bundle built
    with these exact settings, run from a directory with no node_modules, transforms
    successfully once the variable is set and the executable sits beside it.

Known limitations, stated rather than discovered

  • Covers packages that locate an artifact via an environment variable. It does not cover
    better-sqlite3-style packages that compute an addon path at runtime through bindings or
    node-gyp-build — there is no single variable to set for those.
  • One environment variable and one output name means one version per package name.
  • Build platform must equal run platform. libnest already assumes this
    (prismaPlugin calls getBinaryTargetForCurrentPlatform), so this inherits the constraint
    rather than adding it — but it does break under cross-compilation.

Opened from a fork, as I do not have push access to this repository.

🤖 Generated with Claude Code

…bundled

`buildProd` collapses the dependency graph into a single file, which breaks any
package that loads a native artifact through a path relative to its own source.
`prismaPlugin` already solves this — locate the artifact, emit it into the output
directory, point the package back at it with an environment variable — but the
mechanism is welded to one package name and there is no way to declare a second.

esbuild is the case that motivated this. It not only fails to find its executable
once bundled, it actively refuses to run, throwing "The esbuild JavaScript API
cannot be bundled" after comparing __filename against its own expected layout.
Setting ESBUILD_BINARY_PATH both locates the executable and suppresses that
refusal. Verified end to end: a bundle built with these settings, run from a
directory with no node_modules, transforms successfully once the variable is set
and the executable sits beside it.

Consumers now write:

    build: { nativeDependencies: ['esbuild'] }

and the executable is emitted next to the bundle, so it travels with whatever
already copies the output directory. A consumer needing a package without a
built-in recipe supplies `locate`, `outputName`, `packageName` and
`runtimeEnvVar` directly.

Two details worth review attention:

The artifact is located through a `require` rooted at the module that imported
the package during this build, not through `createRequire(import.meta.url)` as
`prismaPlugin` does. That is correct for Prisma, whose engines are a libnest
dependency, but wrong in general: an application frequently resolves a different
version of a transitive dependency than libnest resolves for itself, and pairing
an artifact with a different version of its own JavaScript fails at runtime.
Hooking `onResolve` takes the answer from the actual graph walk instead of
guessing. Resolution is observed, never altered.

Declaring a dependency the application never imports fails the build in `onEnd`,
rather than shipping a bundle whose banner points at a file that was never
emitted. That failure mode is otherwise quiet: a bad path makes esbuild warn and
fall through to normal resolution, which then reports a missing module and blames
the wrong thing entirely.

Also fixes two prerequisites found on the way:

- `prismaPlugin` appended its banner statement without a trailing semicolon. The
  banner concatenates statements from every plugin that appends to it with no
  separator, so a second appending plugin produced a bundle that did not parse.
  There is no newline for ASI to work with.
- `load.ts` validated an `esbuildOptions` field that nothing has read since it was
  removed from the user-facing surface in a987e72. A consumer could set it, have
  it validate cleanly, and have it silently ignored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (79c7ef7) to head (5557ed6).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #76   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           60        62    +2     
  Lines          704       738   +34     
  Branches       118       125    +7     
=========================================
+ Hits           704       738   +34     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@joshunrau
joshunrau merged commit 080794b into DouglasNeuroInformatics:main Aug 4, 2026
3 checks passed
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 8.4.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants