Summary
packages/core/src/state contains two independent implementations of mapWithConcurrency. Neither is reachable from the package root, so consumers who need bounded I/O fan-out have to write their own — which is how this keeps happening.
|
state/job-index.ts:166 |
state/utils/concurrency.ts:17 |
clamps limit |
❌ Math.min(limit, items.length) only |
✅ Math.max(1, Math.min(Math.floor(limit), items.length)) |
| early-returns on empty input |
❌ |
✅ |
| documented |
one line |
full JSDoc |
| tests |
none |
state/utils/__tests__/concurrency.test.ts (order, cap, rejection, limit: 0) |
| used by |
job-metadata.ts:10, 402 |
session-discovery.ts:37, 672 |
Latent bug in the job-index copy
With no lower clamp, limit <= 0 spawns zero workers:
const workers = Array.from({ length: Math.min(limit, items.length) }, worker);
await Promise.all(workers);
return results; // new Array(items.length) — all holes
Promise.all([]) resolves immediately and the function returns an array of undefined, having never called fn. In refreshJobIndex that would silently produce an index of empty entries rather than throwing.
Not reachable today — both call sites pass module constants (IO_CONCURRENCY = 32, HYDRATE_CONCURRENCY = 32) — but it is the kind of thing that becomes reachable the moment someone makes concurrency configurable. utils/concurrency.ts already handles it and has the test for it (concurrency.test.ts:64).
Export surface
Neither copy reaches the package root:
state/utils/index.ts re-exports only ./atomic.js, ./path-safety.js, ./reads.js — concurrency.js is missing from the barrel.
state/index.ts:34 re-exports exactly one job-index symbol, clearJobIndexCache. Not mapWithConcurrency, not refreshJobIndex, not JobIndexRecord/JobIndexEntry/JobIndexRefresh/RefreshJobIndexOptions.
packages/core/package.json has no exports field (only main/types/files), so a deep @herdctl/core/dist/state/job-index.js import does resolve — but that reaches into dist and is not a supported entry point.
Proposal
- Delete the copy in
job-index.ts; import from state/utils/concurrency.js.
- Add
export * from "./concurrency.js" to state/utils/index.ts so it reaches the package root.
- Downstream, Paddock needs exactly this for its own unbounded
Promise.all fan-outs (see edspencer/paddock — /chats/usage opens one read stream per chat, 1,515 concurrently on a production-scale instance, measured at 706 MB peak RSS vs 178 MB at concurrency 16 with no latency cost).
Small, low-risk, and unblocks a downstream fix.
Summary
packages/core/src/statecontains two independent implementations ofmapWithConcurrency. Neither is reachable from the package root, so consumers who need bounded I/O fan-out have to write their own — which is how this keeps happening.state/job-index.ts:166state/utils/concurrency.ts:17limitMath.min(limit, items.length)onlyMath.max(1, Math.min(Math.floor(limit), items.length))state/utils/__tests__/concurrency.test.ts(order, cap, rejection,limit: 0)job-metadata.ts:10, 402session-discovery.ts:37, 672Latent bug in the job-index copy
With no lower clamp,
limit <= 0spawns zero workers:Promise.all([])resolves immediately and the function returns an array ofundefined, having never calledfn. InrefreshJobIndexthat would silently produce an index of empty entries rather than throwing.Not reachable today — both call sites pass module constants (
IO_CONCURRENCY = 32,HYDRATE_CONCURRENCY = 32) — but it is the kind of thing that becomes reachable the moment someone makes concurrency configurable.utils/concurrency.tsalready handles it and has the test for it (concurrency.test.ts:64).Export surface
Neither copy reaches the package root:
state/utils/index.tsre-exports only./atomic.js,./path-safety.js,./reads.js—concurrency.jsis missing from the barrel.state/index.ts:34re-exports exactly one job-index symbol,clearJobIndexCache. NotmapWithConcurrency, notrefreshJobIndex, notJobIndexRecord/JobIndexEntry/JobIndexRefresh/RefreshJobIndexOptions.packages/core/package.jsonhas noexportsfield (onlymain/types/files), so a deep@herdctl/core/dist/state/job-index.jsimport does resolve — but that reaches intodistand is not a supported entry point.Proposal
job-index.ts; import fromstate/utils/concurrency.js.export * from "./concurrency.js"tostate/utils/index.tsso it reaches the package root.Promise.allfan-outs (see edspencer/paddock —/chats/usageopens one read stream per chat, 1,515 concurrently on a production-scale instance, measured at 706 MB peak RSS vs 178 MB at concurrency 16 with no latency cost).Small, low-risk, and unblocks a downstream fix.