Skip to content

Commit a1f00e6

Browse files
Brooooooklynclaude
andauthored
fix(platform): stop large Worker uploads from timing out (#601)
## Problem Large apps cannot deploy to Void Cloud. The `napi-image` deploy ([run](https://github.com/Brooooooklyn/Image/actions/runs/40348054155/job/121667923309)) failed with `The Worker upload did not finish within 120 seconds. Deploy again.` ``` queue job: PUT script to Cloudflare, 120 s timer per attempt body = hdr ─ GET module 1 from R2 ─ bytes ─ hdr ─ GET module 2 ─ ... ─ GET module 744 (one GET at a time, inside the timer) attempt 1: 120 s ✗ → attempt 2: 120 s ✗ → attempt 3 ✗ → attempt 4 ✗ (≈ 490 s, then fail) ``` - The app had 744 modules (413 JS + 331 source maps). 120 s / 744 leaves about 161 ms for each R2 GET, its hash, and its send. - Each retry reads all modules again, so a retry cannot succeed. - The 120 s limit was only measured with local Miniflare R2. - Before #439, the modules were read before the upload, with no timer. That is why this app deployed on 10-07. `@void/md` has a separate problem: `markdown-it-attrs@5` loads `markdown-it` at runtime, but `@void/md` does not provide it. Apps fail with `Cannot find module 'markdown-it'` unless they add it themselves. ## What this PR changes ``` body: hdr ─ module i (already open) ─ bytes ─ open the next modules ahead ... up to 4 modules / 8 MiB ahead, in order timer: no bytes for 60 s → stop (a stalled upload) no answer 120 s after body → stop 8 min for all attempts → stop (shared budget, not 120 s × 4) ``` ``` queued run (15 min Cloudflare limit) ├─ 0 ─────────────────────── 10 min ─────── 15 min │ app upload + DO replays │ compensation reserve │ all end at this deadline ──┘ └─ a DO replay does not start if undoing the hosts already moved would not fit ``` - `attempt_failed` and the new `attempt_done` logs record bytes sent, time waiting on R2, and time waiting on Cloudflare. A later timeout shows which side was slow. - The custom Durable Object host replay sends the same modules. It uses the same read-ahead and limits. - `@void/md` now depends on `markdown-it`. ## Decisions | Question | Choice | Why | |---|---|---| | Read-ahead bound | 4 modules, 8 MiB declared size | One API isolate has 128 MiB for all requests. With the PUT, 4 GETs stay under the 6-connection limit. | | A module over 8 MiB | Opens only at its turn | Opens stay in order. Memory stays bounded. | | Timer model | Idle 60 s + response 120 s + shared 8 min budget | A slow upload that still sends bytes finishes. A stalled one still stops quickly. | | DO replay limit | min(8 min, run deadline), with time kept to undo | The old fixed 60 s would bring the large-bundle timeout back. | | Retry rules | Not changed | 4xx is still final. The saved-script check is still used. | ## Rollout ``` merge → deploy hosted production (platform) → Image redeploys (Brooooooklyn/Image also stops emitting source maps) ``` No CLI change is needed. The fix is all on the platform side. ## Known limits - The real time split on production R2 is not measured yet. The new logs will show it after the first large deploy. - Recovery after an expired lease (`settleExpiredDeployment`) and HTTP rollback still have no run deadline. Each replay there keeps the 8-minute cap. - A Durable Object replay runs inside the cutover transaction, which locks the project row. A slow but progressing replay can now hold that lock for longer than the old 60 s. - Compensation still has no deadline of its own. If restoring the previous release onto a moved Durable Object host takes much longer than the forward move, the queued run can still pass 15 minutes. Lease recovery (`settleExpiredDeployment`) then restores the host, so mixed versions can last until the 18-minute lease expires. This design existed before this PR. A full fix needs a restore budget before any host changes, plus recovery that resumes across runs. It is left for a separate change. ## Testing | Check | Result | |---|---| | `test/unit/worker-upload.test.ts` (read-ahead order, byte bound, release, watchdog, run deadline) | pass | | `test/integration/api-deploy.test.ts` (watchdog cuts stalled attempts, budget stops retries) | pass | | `test/integration/managed-durable-objects.test.ts` (queued run ends DO replay at the deadline; undo time kept) | pass | | `test/integration/deployment-execution.test.ts` (queued upload capped by the run deadline) | pass | | Mutation checks: read-ahead 0, no byte bound, no idle re-arm, no budget, no deadline wiring | each fails a test | | `@void/md` packed into a fresh Yarn PnP project | `markdown-it-attrs` loads; it failed before | | `vp run check` | 33/33 | | `vp run test --base origin/main` | 14911 passed, 28 skipped | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 879ddaa commit a1f00e6

14 files changed

Lines changed: 1322 additions & 86 deletions

‎packages/md/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@
4545
"@shikijs/transformers": "^4.5.0",
4646
"gray-matter": "^4.0.3",
4747
"markdown-exit": "^1.3.0",
48+
"markdown-it": "^15.0.3",
4849
"markdown-it-anchor": "^10.0.0",
4950
"markdown-it-attrs": "^5.0.1",
5051
"markdown-it-container": "^4.0.0",

‎platform/packages/api/src/deploy/pipeline.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -996,6 +996,7 @@ export async function runDeploymentPipeline(input: {
996996
set workerUploaded(value: boolean) {
997997
workerUploaded = value;
998998
},
999+
deadline: execution?.deadline,
9991000
});
10001001
const workerBindings = uploadDeploymentWorkerResult.workerBindings;
10011002
const assetConfig = uploadDeploymentWorkerResult.assetConfig;
@@ -1111,6 +1112,7 @@ export async function runDeploymentPipeline(input: {
11111112
workerFiles,
11121113
migrationsApplied,
11131114
finishOrphanSweep,
1115+
deadline: execution?.deadline,
11141116
});
11151117
} catch (err) {
11161118
return await recoverFailedDeployment({

‎platform/packages/api/src/deploy/publish-worker.ts‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,8 @@ export async function publishWorkerDeployment(input: {
112112
readonly workerFiles: ReadonlyArray<WorkerModule>;
113113
readonly migrationsApplied: number;
114114
readonly finishOrphanSweep: () => Promise<void>;
115+
/** When the uploads of a queued deployment run must end (epoch ms); undefined outside one. */
116+
readonly deadline: number | undefined;
115117
}) {
116118
const {
117119
projectId,
@@ -388,7 +390,15 @@ export async function publishWorkerDeployment(input: {
388390
zeroTrust,
389391
});
390392
const cutoverDomains = await getDomainsByProjectId(txDb, projectId);
391-
await reconcileManagedDurableObjects(context.env, txDb, projectId, deployment.id, manifest);
393+
await reconcileManagedDurableObjects(
394+
context.env,
395+
txDb,
396+
projectId,
397+
deployment.id,
398+
manifest,
399+
undefined,
400+
input.deadline,
401+
);
392402
await putTrackedRouting(projectRouteKey(project), routingJson);
393403
input.routingSwitched = true;
394404
input.logger = input.logger.with({ routingSwitched: true });

‎platform/packages/api/src/deploy/upload-worker.ts‎

Lines changed: 103 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,18 @@ import { RESERVED_BINDING_NAMES } from '../reserved-bindings.ts';
3535
import { deployments } from '../schema.ts';
3636
import { type DeployManifest, type DeployEvent, type Deployment, type Project } from '../types.ts';
3737
import {
38+
WORKER_UPLOAD_BUDGET_MS,
39+
WORKER_UPLOAD_IDLE_MS,
40+
WORKER_UPLOAD_MIN_ATTEMPT_MS,
41+
WORKER_UPLOAD_RESPONSE_MS,
3842
WorkerModuleIntegrityError,
43+
uploadWatchdog,
3944
withKnownLength,
4045
workerUploadBody,
46+
workerUploadDeadline,
47+
type TrackedStream,
48+
type UploadTimeout,
49+
type UploadWatchdog,
4150
type WorkerModule,
4251
} from '../worker-upload.ts';
4352
import {
@@ -57,8 +66,52 @@ import {
5766
} from './shared.ts';
5867
import { platformAppUploadUserAgent } from './user-agent.ts';
5968

60-
// Cloudflare sets no subrequest time limit; a timed-out attempt is retried like a network error.
61-
const WORKER_UPLOAD_ATTEMPT_TIMEOUT_MS = 120_000;
69+
const UNEXPECTED_UPLOAD_RESPONSE = 'worker upload returned an unexpected Cloudflare response';
70+
71+
function uploadTimeoutMessage(
72+
kind: UploadTimeout | undefined,
73+
watchdog: UploadWatchdog,
74+
runLimited: boolean,
75+
): string {
76+
switch (kind) {
77+
case 'idle':
78+
return watchdog.stalledOn() === 'storage'
79+
? `The Worker upload stalled: no module bytes were read from storage for ${WORKER_UPLOAD_IDLE_MS / 1000} seconds. Deploy again.`
80+
: `The Worker upload stalled: Cloudflare accepted no bytes for ${WORKER_UPLOAD_IDLE_MS / 1000} seconds. Deploy again.`;
81+
case 'response':
82+
return `Cloudflare did not answer the Worker upload within ${WORKER_UPLOAD_RESPONSE_MS / 1000} seconds after receiving it. Deploy again.`;
83+
case 'deadline':
84+
return runLimited
85+
? 'The deployment reached its time limit during the Worker upload. Deploy again.'
86+
: `The Worker upload did not finish within ${WORKER_UPLOAD_BUDGET_MS / 60_000} minutes. Deploy again.`;
87+
default:
88+
return 'The Worker upload timed out. Deploy again.';
89+
}
90+
}
91+
92+
/** Where an upload attempt got to, for its log records. Counts and times only. */
93+
function uploadFields(
94+
upload: TrackedStream & Readonly<{ length: number }>,
95+
watchdog: UploadWatchdog,
96+
attemptStart: number,
97+
budgetEndsAt: number,
98+
): Record<string, unknown> {
99+
const progress = upload.progress();
100+
const timeout = watchdog.fired() ?? null;
101+
return {
102+
timeout,
103+
// Not `phase`: the logger keeps that key for the deploy step.
104+
uploadPhase: progress.done ? 'response' : 'sending',
105+
...(timeout === 'idle' ? { stalledOn: watchdog.stalledOn() } : {}),
106+
bytesSent: progress.bytes,
107+
bodyBytes: upload.length,
108+
modulesStarted: progress.sourcesStarted,
109+
modules: progress.sourcesTotal,
110+
sourceWaitMs: progress.sourceWaitMs,
111+
sendMs: progress.doneAt === undefined ? null : progress.doneAt - attemptStart,
112+
budgetRemainingMs: budgetEndsAt - Date.now(),
113+
};
114+
}
62115

63116
/** The Cloudflare error code of a rejected replay of a saved upload, if the body has one. */
64117
function savedUploadReplayCode(body: string): number | undefined {
@@ -126,6 +179,8 @@ export async function uploadDeploymentWorker(input: {
126179
readonly owner: ApiEnv['Variables']['user'];
127180
workerUploadUrl: string | undefined;
128181
workerUploaded: boolean;
182+
/** When the uploads of a queued deployment run must end (epoch ms); undefined outside one. */
183+
readonly deadline: number | undefined;
129184
}) {
130185
const {
131186
markPhase,
@@ -490,12 +545,28 @@ export async function uploadDeploymentWorker(input: {
490545
}
491546
return saved;
492547
};
548+
// Cloudflare sets no subrequest time limit. An attempt is cut when it stops
549+
// moving (WORKER_UPLOAD_IDLE_MS, WORKER_UPLOAD_RESPONSE_MS) and retried like
550+
// a network error; all attempts together share WORKER_UPLOAD_BUDGET_MS, cut
551+
// short by the end of a queued deployment run. Another attempt starts only
552+
// when at least WORKER_UPLOAD_MIN_ATTEMPT_MS is left after its backoff.
553+
const budgetEndsAt = workerUploadDeadline(input.deadline);
554+
const runLimited = budgetEndsAt === input.deadline;
493555
await retryWithBackoff(
494556
async (attempt) => {
495557
const attemptStart = Date.now();
496-
const lastAttempt = attempt === CF_RETRY.maxRetries;
497558
// A streamed body is sent once, so each attempt reopens the modules.
498559
const upload = workerUploadBody(metadata, workerFiles);
560+
const watchdog = uploadWatchdog(upload.progress, {
561+
idleMs: WORKER_UPLOAD_IDLE_MS,
562+
responseMs: WORKER_UPLOAD_RESPONSE_MS,
563+
deadline: budgetEndsAt,
564+
});
565+
// No retry once the attempts so far leave too little budget for another.
566+
const isFinal = () =>
567+
attempt === CF_RETRY.maxRetries ||
568+
watchdog.fired() === 'deadline' ||
569+
budgetEndsAt - Date.now() < WORKER_UPLOAD_MIN_ATTEMPT_MS + CF_RETRY.delayMs * 2 ** attempt;
499570
let cfResponse: Response | undefined;
500571
let cfBody: string;
501572
// The request and its response body are retried together: a connection
@@ -505,7 +576,7 @@ export async function uploadDeploymentWorker(input: {
505576
method: 'PUT',
506577
headers: { ...cfHeaders, 'Content-Type': upload.contentType },
507578
body: withKnownLength(upload.stream, upload.length),
508-
signal: AbortSignal.timeout(WORKER_UPLOAD_ATTEMPT_TIMEOUT_MS),
579+
signal: watchdog.signal,
509580
});
510581
cfBody = await cfResponse.text();
511582
} catch (error) {
@@ -516,7 +587,9 @@ export async function uploadDeploymentWorker(input: {
516587
if (failure instanceof WorkerModuleIntegrityError) {
517588
throw failure;
518589
}
519-
const timedOut = (error as { name?: unknown } | null)?.name === 'TimeoutError';
590+
const timeout = watchdog.fired();
591+
const timedOut =
592+
timeout !== undefined || (error as { name?: unknown } | null)?.name === 'TimeoutError';
520593
const cause = failure ?? error;
521594
const fields = {
522595
attempt,
@@ -525,6 +598,7 @@ export async function uploadDeploymentWorker(input: {
525598
code: timedOut ? 'timeout' : 'transport',
526599
status: cfResponse?.status,
527600
err: cause instanceof Error ? cause.message : String(cause),
601+
...uploadFields(upload, watchdog, attemptStart, budgetEndsAt),
528602
};
529603
// A 4xx stays final even when its body drops, unless it replays an
530604
// upload that was saved. Any other failure is retried.
@@ -534,25 +608,26 @@ export async function uploadDeploymentWorker(input: {
534608
}
535609
throw new Error(`worker upload failed (HTTP ${cfResponse.status})`, { cause: error });
536610
}
537-
if (lastAttempt) {
611+
if (isFinal()) {
538612
// This attempt or an earlier one may have been saved.
539613
if (await savedDespiteFailure(fields)) {
540614
return true;
541615
}
542-
// The timer also covers the module reads that feed the body.
616+
// The watchdog also covers the module reads that feed the body.
543617
throw (
544618
failure ??
545619
(timedOut
546-
? new Error(
547-
`The Worker upload did not finish within ${WORKER_UPLOAD_ATTEMPT_TIMEOUT_MS / 1000} seconds. Deploy again.`,
548-
{ cause: error },
549-
)
620+
? new Error(uploadTimeoutMessage(timeout, watchdog, runLimited), { cause: error })
550621
: error)
551622
);
552623
}
553624
maybeSaved = true;
554625
logger.warn('worker_upload', 'attempt_failed', fields);
555626
return undefined;
627+
} finally {
628+
// Frees the storage reads of a failed attempt before the backoff.
629+
watchdog.dispose();
630+
upload.release();
556631
}
557632

558633
if (!cfResponse.ok) {
@@ -562,10 +637,11 @@ export async function uploadDeploymentWorker(input: {
562637
latencyMs: Date.now() - attemptStart,
563638
status: cfResponse.status,
564639
bodyPreview: cfBody.slice(0, 500),
640+
...uploadFields(upload, watchdog, attemptStart, budgetEndsAt),
565641
};
566642
// Only 5xx is retried. A 429 stays final: the install shares one API
567643
// token, and Cloudflare blocks it for minutes once it hits the limit.
568-
if (cfResponse.status >= 500 && !lastAttempt) {
644+
if (cfResponse.status >= 500 && !isFinal()) {
569645
maybeSaved = true;
570646
logger.warn('worker_upload', 'attempt_failed', fields);
571647
return undefined;
@@ -593,14 +669,28 @@ export async function uploadDeploymentWorker(input: {
593669
latencyMs: Date.now() - attemptStart,
594670
code: 'cf_plain_text',
595671
bodyPreview: cfBody.slice(0, 100),
672+
...uploadFields(upload, watchdog, attemptStart, budgetEndsAt),
596673
});
674+
if (isFinal()) {
675+
throw new Error(UNEXPECTED_UPLOAD_RESPONSE);
676+
}
597677
return undefined;
598678
}
679+
const progress = upload.progress();
680+
logger.info('worker_upload', 'attempt_done', {
681+
attempt,
682+
latencyMs: Date.now() - attemptStart,
683+
bodyBytes: upload.length,
684+
modules: progress.sourcesTotal,
685+
sendMs: progress.doneAt === undefined ? null : progress.doneAt - attemptStart,
686+
sourceWaitMs: progress.sourceWaitMs,
687+
responseMs: progress.doneAt === undefined ? null : Date.now() - progress.doneAt,
688+
});
599689
return true;
600690
},
601691
{
602692
...CF_RETRY,
603-
errorMessage: 'worker upload returned an unexpected Cloudflare response',
693+
errorMessage: UNEXPECTED_UPLOAD_RESPONSE,
604694
},
605695
);
606696
input.workerUploaded = true;

‎platform/packages/api/src/deployment-execution.ts‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,16 @@ export type {
1818

1919
// Queue invocations have a 15-minute wall-clock limit. Recovery starts after
2020
// that invocation is gone, and never repeats its provider writes or SQL.
21+
export const DEPLOYMENT_INVOCATION_MS = 15 * 60 * 1000;
2122
export const DEPLOYMENT_EXECUTION_LEASE_MS = 18 * 60 * 1000;
23+
// Worker uploads of a run (the application, then each custom Durable Object
24+
// host) end this long before the invocation does. Each host upload also leaves
25+
// time before then to replay the committed release onto every host the run has
26+
// moved, so a failed run can undo them. This reserve then covers restoring
27+
// routing, queues and cron, and an undo slower than the move it reverts. The
28+
// other 10 minutes cover the 8-minute application upload budget and the steps
29+
// before it.
30+
export const DEPLOYMENT_COMPENSATION_RESERVE_MS = 5 * 60 * 1000;
2231
const MAX_PROGRESS_EVENTS = 1000;
2332

2433
export async function installationDeploymentAdmissionClosed(
@@ -546,6 +555,7 @@ export async function processDeploymentBatch(
546555
batch: MessageBatch<unknown>,
547556
env: ApiEnv['Bindings'],
548557
): Promise<void> {
558+
const uploadDeadline = Date.now() + DEPLOYMENT_INVOCATION_MS - DEPLOYMENT_COMPENSATION_RESERVE_MS;
549559
const db = createDb(env.HYPERDRIVE.connectionString);
550560
for (const message of batch.messages) {
551561
const body = message.body as { deploymentId?: unknown } | null;
@@ -596,7 +606,7 @@ export async function processDeploymentBatch(
596606
);
597607
} else {
598608
const { executeQueuedDeployment } = await import('./routes/deploy.ts');
599-
await executeQueuedDeployment(env, db, operation);
609+
await executeQueuedDeployment(env, db, operation, uploadDeadline);
600610
}
601611
const latest = await getDeploymentOperation(db, id);
602612
if (latest?.status !== 'terminal') {

‎platform/packages/api/src/env.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,8 @@ export type ApiEnv = Env & {
232232
deploymentExecution?: {
233233
operation: import('./deployment-operation-types.ts').DeploymentOperation;
234234
workerFiles: Array<import('./worker-upload.ts').WorkerModule>;
235+
/** When this run's Worker uploads must end (epoch ms), leaving time to compensate. */
236+
deadline: number;
235237
};
236238
};
237239
};

0 commit comments

Comments
 (0)