perf(fuse/backend): PATCH on large mostly-unchanged file is slow — parallelize sequential retained-part UploadPartCopy (16s → ~1-2s)

[qiffang]

Problem

A PATCH /v1/fs/.../output/douyin/video_720.mp4 took duration_ms=16060 (~16s), status 202.

Log signal for that trace:

• new_size≈308MB, total_parts=37, dirty_parts=1, copied_parts=36
• i.e. only one ~8MB part actually changed; the other 36 retained parts (~288MB) must be re-included in the new multipart object.

Root cause

In pkg/backend/patch.go, the plan-execution loop that re-assembles the retained parts is sequential:

for _, p := range newParts {
    // unchanged part → server-side copy
    _, err := b.s3.UploadPartCopy(ctx, newS3Key, mpu.UploadID, p.Number, sourceKey, partStart, partEnd)
    ...
}

No goroutine / errgroup / semaphore. So the 36 UploadPartCopy calls run one after another: 16060ms / 36 ≈ 446ms per copy (normal for an ~8MB S3 server-side part-copy) → 36 sequential = ~16s. The latency is the retained-part copies serialized, not the dirty-data transfer (only 1 part).

The get_upload_reservation / get_upload_by_path not_found warns on the same trace are benign first-patch misses (~60ms), not the latency source.

The 202 is written after InitiatePatchUploadIfRevision (server.go:2444 → 2496) completes the copies — it returns a PatchPlan (presigned dirty-part upload URL + read URLs); the client then uploads the dirty part and calls confirm (ConfirmUploadWithTags) which does CompleteMultipartUpload. So 202 is not a durability signal; there is already an upload+confirm step after it.

Fix — Option A (this issue): parallelize the in-request UploadPartCopy loop

Bounded-concurrency the copy loop. Contract unchanged, pure server-side. Expected 16s → ~1–2s.

Acceptance gates (from dat9-dev1 + adversary-1 + adversary-2):

• Bounded concurrency (semaphore, e.g. 8–16) — NOT unbounded fan-out (S3 UploadPartCopy has per-prefix rate limits).
• Retry on 503 SlowDown.
• Part-number + ETag ordering correct in the completed multipart (copies may finish out of order).
• Failure atomicity: first error cancels remaining in-flight copies, AbortMultipartUpload runs exactly once, no leaked parts / MPU.
• ctx cancellation stops subsequent copies.
• Discriminating regression + benchmark with many copied_parts proving the serial bottleneck is broken (assert concurrency, first-error abort, no leak).

Follow-up — Option B (separate issue, not this one): async copy

Since 202 returns a plan and a confirm step already follows, the retained-part copies only need to complete before CompleteMultipartUpload at confirm-time. So the handler could return the plan immediately, run copies in a background window, and join/wait at confirm — driving the handler sub-second without changing 202 semantics. More complex (copy-vs-confirm join + failure propagation); evaluate separately.

Owner: dat9-dev1 (Option A). Reviewers: adversary-1, adversary-2. drive9 is qiffang-owned → PR + regression, merge on qiffang's signal.

[qiffang]

Assignment update: qiffang assigned Option A implementation to dat9-dev2 (2026-07-31), with emphasis on correctness. dat9-dev1 (analysis author) reverts to Dev-Lead review alongside adversary-1/adversary-2. All gates above stand; Option B stays a separate follow-up (not in this PR).

[qiffang]

Implementation moved to @dat9-dev2 under Raft task #202 after the issue was opened. PR #821 implements Option A only at exact main@41503f74493abd4c48ffe3085d7890dcc31a056c:

• fixed request-local concurrency cap of 8
• AWS SDK standard retryer remains the single 503 / SlowDown retry layer
• first copy error or request cancellation stops scheduling and cancels peers
• waits for every worker before one detached, bounded multipart abort
• no reservation/upload metadata exists on copy failure
• CopiedParts is derived from original part order; multipart ETags remain S3-owned and ListParts-sorted

Local deterministic unit/race/compile/vet gates pass. The 36-part simulated-latency benchmark improves from 46.23ms/op at concurrency 1 to 6.44ms/op at concurrency 8 (~7.2x). DB-backed LocalS3 integration regressions compile locally and will run in hosted CI because this machine lacks rootless Docker.

[qiffang]

COS follow-up acceptance point (non-blocking for PR #821):

• run a Tencent COS endpoint integration smoke through the production newTencent -> AWSS3Client path, not LocalS3
• seed a multipart-backed object, PATCH with ~36 retained/copied parts and a low dirty-part ratio, then verify exact bytes and part ordering after confirm
• record total PATCH-init latency, per-copy latency, maximum in-flight copies, retry / 503 / SlowDown counts, and abort/leaked-MPU state
• repeat enough times to compare the previous serial path with min(8, copied_parts) under the same bucket/prefix/network conditions
• if COS throttling materially dominates at 8, open a separate configuration change for a per-backend concurrency cap; do not fold that into Option A / PR perf(backend): parallelize retained PATCH copies #821

This is a real-COS performance/compatibility validation gate. Functional applicability is already established because Tencent configuration uses the shared AWSS3Client.UploadPartCopy; exact COS latency and throttle limits are deployment facts, not assumed equivalent to AWS S3.