perf: stop lockfile reads after the environment document

[MindTooth]

Summary

Stop reading pnpm-lock.yaml once the chunk containing the leading environment-document separator has been read. This is a follow-up to #72: that change already avoids parsing the dependency document, but readFileSync(..., 'utf8') still reads and decodes the complete lockfile and normalizes every CRLF pair before extracting the small environment document.

For a large dependency lockfile with a small environment document, the lookup now reads 64 KiB instead of the entire file. For example, the generated 53.47 MiB LF fixture went from 7.127 ms and 107.98 MiB added peak RSS to 0.095 ms and 0.94 MiB. The real n8n dependency document, wrapped with a controlled environment prefix, went from 0.494 ms to 0.113 ms.

The main justification is avoiding allocations and reads proportional to dependency-document size. These are lookup benchmarks, not a claim about total setup-step time; downloading and installing pnpm usually dominates that time.

Implementation

• Replace the whole-file read with a synchronous openSync / readSync / closeSync loop, preserving the synchronous caller.
• Reuse one 64 KiB buffer and use Node's StringDecoder to handle UTF-8 characters split across reads.
• Search the current decoded window for literal LF/CRLF separator lines, retaining six characters of overlap for separators split across reads.
• Accumulate decoded chunks and join them once at the boundary or EOF. This avoids repeatedly rescanning or normalizing an accumulated large environment document.
• Close the file in finally, including early returns and read failures.
• Reuse the existing extractEnvDocument and YAML parsing code. There is no document-size cap.
• Regenerate dist/index.js; the bundle grows by 503 bytes. No dependencies are added.

The reader may include dependency data from the final 64 KiB chunk. Files without an environment header return early; env-only files must still be read to EOF. Cost therefore scales with environment-document size plus at most one read chunk, rather than the complete dependency document.

Compatibility and pnpm comparison

The existing behavior is retained for:

• UTF-8 BOM, LF, CRLF, and boundaries split across reads;
• missing lockfiles and files without an environment document;
• env-only lockfiles, including a closing separator without a final newline;
• malformed environment YAML, which still throws;
• malformed dependency YAML, which is still ignored for version selection;
• exact pins and dist-tags, which bypass lockfile selection;
• changed manifest ranges and prerelease selection (includePrerelease: true).

pnpm's own reader also uses 64 KiB reads, StringDecoder, and early exit on a literal document separator. This implementation keeps setup's synchronous API and existing extractor rather than importing a new dependency or copying the entire upstream helper.

Upstream pnpm additionally requires a root-importer configDependencies marker when identifying environment-only documents. This is a literal text check, not comprehensive schema validation. Setup intentionally preserves its existing, more permissive version-selection behavior. Aligning these rules is outside the scope of this performance change.
Document boundaries continue to follow pnpm's literal separator handling rather than general YAML syntax. Separator comments, indentation, and bare CR line endings are not supported.

Benchmark results

Each pair is before → after. RSS is the increase in process peak resident memory measured around one lookup, not total process memory.

Input	Time (ms/lookup)	Bytes returned by file APIs	Added peak RSS (MiB)
Setup lockfile + env, 31.9 KiB	0.100 → 0.112	32,677 → 32,677	0.84 → 0.72
pnpm monorepo + env, 0.814 MiB	0.191 → 0.108	853,246 → 65,536	1.86 → 0.77
n8n monorepo + env, 1.583 MiB	0.494 → 0.113	1,660,007 → 65,536	3.56 → 0.98
Env-only, 134 bytes	0.091 → 0.100	134 → 134	1.08 → 0.86
Generated LF, 1.051 MiB	0.174 → 0.097	1,101,839 → 65,536	2.48 → 0.88
Generated LF, 10.609 MiB	1.229 → 0.097	11,123,829 → 65,536	22.09 → 0.80
Generated LF, 53.466 MiB	7.127 → 0.095	56,062,779 → 65,536	107.98 → 0.94
No env header, LF, 53.465 MiB	6.981 → 0.022	56,062,640 → 65,536	107.20 → 0.23
Generated CRLF, 11.017 MiB	25.510 → 0.139	11,551,831 → 65,536	75.56 → 0.86
Env-only stress case, 10.000 MiB	1.626 → 2.619	10,485,897 → 10,485,897	20.92 → 31.48

Method

• Apple M4 Pro, macOS arm64, Node v26.11.1.
• Before: base commit a0881977cd8ea73b1ed212236c99ca10dcc80cf4. After: this change. Both readers bundled with the same installed esbuild, YAML, and semver versions.
• Measure the complete readLockedPnpmVersion(directory, '12') lookup, including YAML parsing and range selection; exclude module startup.
• Time: warm filesystem cache; 10 warmups; median of five batches of 30 lookups; GC before each batch.
• Memory: one lookup in a fresh process after module loading and GC. Use the change in process.resourceUsage().maxRSS; also record post-call heap and ArrayBuffer deltas. Post-call heap alone misses transient allocations.
• Bytes: instrument Node file APIs, excluding module loading. These are application bytes, not physical disk traffic or OS readahead.
• Real dependency data: this repository's lockfile, pnpm's lockfile, and n8n's lockfile. Each receives the same controlled 139-byte environment prefix selecting 12.8.1; these projects are not asserted to pin that version.
• Generated dependency documents contain unique package records with resolution and dependency entries. The env-only stress case contains a long YAML comment. Small RSS and microsecond timing differences are noisy.

Larger structured environment documents

An additional comparison used generated documents following pnpm's environment shape (configDependencies, packageManagerDependencies, packages, and snapshots) ahead of n8n's real dependency document. Five warmups and median of three batches of ten lookups were used for these more expensive inputs.

Environment workload	Environment bytes (LF)	LF ms, before → after	CRLF ms, before → after
Small environment	249	0.372 → 0.155	1.445 → 0.140
500 config packages	109,023	13.471 → 13.236	14.590 → 13.296
5,000 config packages, upper stress case	1,098,023	513.779 → 511.745	507.535 → 509.833

Parsing the larger environment dominates both implementations. The 5,000-package case is an upper stress workload, not a claim about typical environment graphs. Its 2,757,900-byte LF file read 1,114,112 bytes with this implementation: exactly the 17 chunks needed to reach its separator at byte 1,098,027.

Tests and validation

All existing tests are retained. Four focused regression tests were added for:

1. LF/CRLF separators at positions across a 64 KiB boundary, with and without BOM.
2. UTF-8 split across reads, including a referenced Unicode YAML anchor; env-only endings; malformed YAML after the first chunk.
3. Reads artificially restricted to 1–8 bytes, split headers/BOM, empty inputs, and invalid or double-BOM headers.
4. Bounded reads for present/absent environment documents and file-descriptor cleanup after a read failure.

Validation completed:

• Install-pnpm suite: 33/33 passed, including 13/13 locked-version tests.
• pnpm exec tsc --noEmit: passed.
• pnpm build: passed; action bundle targets Node 24.
• Temporary differential comparison: 720 cases passed, comparing selected versions or exact error messages for BOM/header/separator/size combinations against the base reader.
• Additional temporary checks: symlinked reads; zero lockfile opens for exact pins/tags; per-project lockfile + inherited range + package.yaml + symlink + BOM/CRLF + registry mirror. Base and changed implementations produced identical results.

The full suite is not green locally: pnpm test reported 103 passed / 4 failed. The unchanged base, archived into a separate directory, reproduced the same four failures (99 passed / 4 failed). Running the changed suite outside the sandbox also reproduced them. They are the src/pnpm-commands.test.mjs cases for:

• native binary execution with a relative destination and separate project directory;
• default cache paths following workspace membership;
• shared-workspace installation using the root lockfile;
• per-project installation using the member lockfile.

The failures involve absent command-fixture output and workspace-root selection. Their precise local cause was not established; they are not attributed to this PR.

Source/test whitespace checks pass. A full git diff --check reports two trailing-whitespace lines in dependency template literals in the regenerated bundle; the original bundle contains the same template-literal whitespace.

Tradeoffs and remaining coverage

• The artificial 10 MiB env-only case is slower and uses about 10.6 MiB more peak RSS because chunks must be retained and joined. Ordinary env-only input changes by roughly 9 microseconds. There is no cap or fallback reread policy added for this unusual workload.
• The bounded-read regression assertion currently covers a small environment followed by a large tail. Multi-chunk structured environments were checked separately, but that byte-count check is not in the permanent suite.
• The combined installer configuration case and registry forwarding were checked separately; the existing permanent resolver stub does not assert its registry argument.
• Windows/Linux execution, Node 24 runtime execution, cold-cache storage performance, and end-to-end action timing were not measured. The build target is Node 24; runtime checks used Node 26.

Reproducing the main benchmark

The following creates before/after reader bundles and benchmark inputs in a temporary directory. Run from this PR checkout with dependencies installed and Node supporting --expose-gc.

Commands and standalone benchmark harness

export BENCH_DIR="$(mktemp -d)"
node --input-type=module - "$BENCH_DIR" <<'JS'
import { build } from 'esbuild'
import { execFileSync } from 'node:child_process'
import path from 'node:path'
const directory = process.argv[2]
await build({
  stdin: {
    contents: execFileSync('git', ['show', 'a0881977cd8ea73b1ed212236c99ca10dcc80cf4:src/install-pnpm/locked-version.ts'], { encoding: 'utf8' }),
    resolveDir: path.resolve('src/install-pnpm'), loader: 'ts',
  },
  bundle: true, platform: 'node', format: 'cjs', outfile: path.join(directory, 'baseline.cjs'),
})
await build({
  entryPoints: ['src/install-pnpm/locked-version.ts'], bundle: true, platform: 'node',
  format: 'cjs', outfile: path.join(directory, 'candidate.cjs'),
})
JS
curl -L --fail 'https://raw.githubusercontent.com/pnpm/pnpm/8b16e74e9688640f355f7938bf973ffb45605866/pnpm-lock.yaml' -o "$BENCH_DIR/pnpm-upstream.yaml"
curl -L --fail 'https://raw.githubusercontent.com/n8n-io/n8n/7424f0ca0452f9dc9fe4cf6627ecbf0b315ed1e3/pnpm-lock.yaml' -o "$BENCH_DIR/n8n-upstream.yaml"

Save the following as $BENCH_DIR/benchmark.mjs, then run node "$BENCH_DIR/benchmark.mjs" from the checkout. Results are printed and written to $BENCH_DIR/results.json.

import fs from 'node:fs'
import path from 'node:path'
import { syncBuiltinESMExports } from 'node:module'
import { spawnSync } from 'node:child_process'
import assert from 'node:assert/strict'

const root = path.dirname(new URL(import.meta.url).pathname)
const [mode, implementation, fixture] = process.argv.slice(2)
if (mode === 'worker') {
  let bytes = 0
  const readFile = fs.readFileSync, read = fs.readSync
  fs.readFileSync = (...args) => {
    const value = readFile(...args)
    if (String(args[0]).endsWith('pnpm-lock.yaml')) bytes += Buffer.byteLength(value)
    return value
  }
  fs.readSync = (...args) => { const n = read(...args); bytes += n; return n }
  syncBuiltinESMExports()
  const { readLockedPnpmVersion } = await import(`${root}/${implementation}.cjs`)
  bytes = 0
  const directory = `${root}/${fixture}`
  if (process.env.MEMORY) {
    global.gc()
    const before = process.memoryUsage(), peak = process.resourceUsage().maxRSS
    const result = readLockedPnpmVersion(directory, '12')
    const after = process.memoryUsage()
    assert.equal(result, fixture.includes('absent') ? undefined : '12.8.1')
    console.log(JSON.stringify({ bytes, heapMiB: (after.heapUsed - before.heapUsed) / 2**20,
      bufferMiB: (after.arrayBuffers - before.arrayBuffers) / 2**20,
      peakRssMiB: (process.resourceUsage().maxRSS - peak) / 1024 }))
  } else {
    for (let i = 0; i < 10; i++) readLockedPnpmVersion(directory, '12')
    const samples = []
    for (let trial = 0; trial < 5; trial++) {
      global.gc()
      const start = performance.now()
      for (let i = 0; i < 30; i++) readLockedPnpmVersion(directory, '12')
      samples.push((performance.now() - start) / 30)
    }
    console.log(JSON.stringify({ ms: samples.sort((a, b) => a - b)[2] }))
  }
} else {
  const env = "---\nlockfileVersion: '9.0'\nimporters:\n  .:\n    packageManagerDependencies:\n      pnpm:\n        specifier: '12'\n        version: 12.8.1\n---\n"
  const entry = "  example@1.0.0:\n    resolution: {integrity: sha512-example}\n    dependencies:\n      other: 2.0.0\n"
  const actual = fs.readFileSync(`${root}/pnpm-upstream.yaml`, 'utf8')
  const cases = {
    'setup-small': env + fs.readFileSync(`${process.cwd()}/pnpm-lock.yaml`, 'utf8'),
    'pnpm-monorepo': env + actual,
    'n8n-monorepo': env + fs.readFileSync(`${root}/n8n-upstream.yaml`, 'utf8'),
    'env-only-small': env.slice(0, -5),
    'synthetic-1MiB': env + 'packages:\n' + entry.repeat(Math.ceil(2**20 / entry.length)),
    'synthetic-10MiB': env + 'packages:\n' + entry.repeat(Math.ceil(10 * 2**20 / entry.length)),
    'synthetic-50MiB': env + 'packages:\n' + entry.repeat(Math.ceil(50 * 2**20 / entry.length)),
    'absent-50MiB': 'packages:\n' + entry.repeat(Math.ceil(50 * 2**20 / entry.length)),
    'env-only-10MiB': env.slice(0, -5) + '\n# ' + 'x'.repeat(10 * 2**20),
  }
  cases['synthetic-10MiB-CRLF'] = cases['synthetic-10MiB'].replace(/\n/g, '\r\n')
  for (const name of Object.keys(cases).filter(name => name.startsWith('synthetic') || name.startsWith('absent'))) {
    let index = 0
    cases[name] = cases[name].replace(/example@/g, () => `example-${index++}@`)
  }
  const rows = []
  for (const [name, content] of Object.entries(cases)) {
    fs.mkdirSync(`${root}/${name}`, { recursive: true })
    fs.writeFileSync(`${root}/${name}/pnpm-lock.yaml`, content)
    for (const impl of ['baseline', 'candidate']) {
      const run = memory => {
        const result = spawnSync(process.execPath, ['--expose-gc', import.meta.filename, 'worker', impl, name],
          { encoding: 'utf8', env: { ...process.env, MEMORY: memory ? '1' : '' } })
        if (result.status !== 0) throw new Error(result.stderr)
        return JSON.parse(result.stdout)
      }
      rows.push({ fixture: name, implementation: impl, fileBytes: Buffer.byteLength(content), ...run(false), ...run(true) })
    }
  }
  fs.writeFileSync(`${root}/results.json`, JSON.stringify(rows, null, 2) + '\n')
  console.table(rows)
}

Summary by CodeRabbit

• Bug Fixes
  • Improved pnpm version detection for large lockfiles and files with unusual formatting or UTF-8 content.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: f040fe6b-a442-4325-8ab7-4555acb2bf02

---

---

📥 Commits

Reviewing files that changed from the base of the PR and between a088197 and d89ad6e.

---

---

⛔ Files ignored due to path filters (1)

• dist/index.js is excluded by !**/dist/**

---

---

📒 Files selected for processing (2)

• src/install-pnpm/locked-version.test.mjs
• src/install-pnpm/locked-version.ts

---

---

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

---

---

📜 Recent review details

⏰ Context from checks skipped due to timeout. (34)

• GitHub Check: install: false skips pnpm install
• GitHub Check: Node version file (tool-versions / ubuntu-latest)
• GitHub Check: Smoke (ubuntu-24.04-arm / pnpm 12.0.0)
• GitHub Check: Explicit runtime overrides devEngines.runtime
• GitHub Check: Workflow-set pnpm_config_global_shims is not overwritten
• GitHub Check: Prime runtime cache
• GitHub Check: Smoke (ubuntu-latest / pnpm 12.0.0)
• GitHub Check: require-lockfile (standalone project / ubuntu-latest)
• GitHub Check: Lockfile verification cache (windows-latest, cache=true)
• GitHub Check: Lockfile verification cache (macos-latest, cache=true)
• GitHub Check: pnpm command execution (ubuntu-latest / pnpm 12.5.1)
• GitHub Check: Installed runtime beats context-aware shims
• GitHub Check: require-lockfile (standalone project / windows-latest)
• GitHub Check: Smoke pnpm 11 (ubuntu-24.04-arm)
• GitHub Check: Runtime deno
• GitHub Check: Smoke pnpm 11 (ubuntu-latest)
• GitHub Check: Lockfile verification cache (ubuntu-latest, cache=true)
• GitHub Check: Lockfile verification cache (ubuntu-latest, cache=false)
• GitHub Check: Smoke (macos-latest / pnpm 12.0.0)
• GitHub Check: Smoke (windows-latest / pnpm 12.0.0)
• GitHub Check: pnpm command execution (windows-latest / pnpm 11.22.0)
• GitHub Check: Workflow-set PNPM_CONFIG_GLOBAL_SHIMS is not overwritten
• GitHub Check: Smoke pnpm 11 (windows-latest)
• GitHub Check: pnpm command execution (windows-latest / pnpm 12.5.1)
• GitHub Check: Runtime bun
• GitHub Check: Smoke pnpm 11 (macos-latest)
• GitHub Check: Runtime node + install on pnpm 11
• GitHub Check: Runtime from devEngines.runtime
• GitHub Check: Runtime node@22 (macos-latest)
• GitHub Check: Runtime node@22 (windows-latest)
• GitHub Check: Node version file (nvmrc / windows-latest)
• GitHub Check: pnpm command execution (ubuntu-latest / pnpm 11.22.0)
• GitHub Check: Cache Windows extended-length store path
• GitHub Check: check-dist

---

---

🧰 Additional context used

🪛 OpenGrep (1.30.1)

src/install-pnpm/locked-version.ts

[ERROR] 55-55: Dynamic command passed to child_process.exec/execSync. Use child_process.execFile or spawn with an argument array instead.

(coderabbit.command-injection.exec-js)

---

---

---

🔇 Additional comments (2)

src/install-pnpm/locked-version.ts (1)

1-1: LGTM!

Also applies to: 4-4, 15-17, 36-66

---

src/install-pnpm/locked-version.test.mjs (1)

5-5: LGTM!

Also applies to: 89-174

---

---

---

---

📝 Walkthrough

📝 Walkthrough

Walkthrough

readLockedPnpmVersion now extracts the lockfile environment document with bounded, streamed reads. Tests cover read boundaries, UTF-8 decoding, document endings, read errors, and file closure.

Changes

Locked Version Lookup

Layer / File(s)	Summary
Stream environment document src/install-pnpm/locked-version.ts, src/install-pnpm/locked-version.test.mjs	readLockedPnpmVersion uses 64 KiB reads to find and decode the environment document. Tests cover boundary cases, short reads, read limits, error propagation, and descriptor closure.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~12 minutes

Sequence Diagram(s)

sequenceDiagram
  participant readLockedPnpmVersion
  participant readEnvDocument
  participant FileSystem
  participant StringDecoder
  readLockedPnpmVersion->>readEnvDocument: request environment document
  readEnvDocument->>FileSystem: open lockfile and read chunks
  readEnvDocument->>StringDecoder: decode UTF-8 chunks
  readEnvDocument->>FileSystem: close file descriptor
  readEnvDocument-->>readLockedPnpmVersion: return document or undefined

Loading

---

---

Merge Risk: ⚪ Minimal · up to d89ad

The change reduces lockfile reads when the pnpm version is looked up and keeps the existing behavior for missing files and parsing. No actionable merge risk remains.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main performance change: stopping lockfile reads after the environment document.

---

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

---

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{I’m a rabbit, ears alert,
I watch each chunk arrive.
The boundary holds no secrets,
UTF-8 stays alive.
The file is closed; I hop away,
With locked versions in my hay.}

_{Comment @coderabbitai help to get the list of available commands.}