fix(media): retry a ranged media read that comes back non-206 - #2014
fix(media): retry a ranged media read that comes back non-206#2014giladresisi wants to merge 2 commits into
Conversation
Cloudflare in front of R2 intermittently ignores Range and answers 200
with the full object. A single such answer failed the whole post. Retry
the identical range up to 3 times before throwing, and record the real
{status, statusText, ok} in the failure details.
Strix Security ReviewNo security issues found. Updated for Reviewed by Strix |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
| } as any); | ||
|
|
||
| // A store that ignores Range (200 with the full file) or answers with an | ||
| // error page would corrupt the upload at this offset. | ||
| // error page would corrupt the upload at this offset, so the body is | ||
| // never used - but the same store answers the identical range correctly | ||
| // seconds later, so retry the read instead of failing the whole upload | ||
| // on one bad answer. | ||
| if (response.status !== 206) { | ||
| if (totalRetries <= 2) { | ||
| await timer(5000); | ||
| return this.youtubeChunkStream(path, start, end, totalRetries + 1); |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Valid, and fixed in 8c4d511. Confirmed at all three read sites: youtubeChunkStream, tiktokChunkStream and the generic SocialAbstract.mediaChunk, which has the same pattern and also gained the retry in this PR.
The reason it matters more here than the wording suggests: the abandoned body in this path is the whole object, since the failure being retried is exactly the case where the CDN ignores Range and answers 200 with the complete file (600-770 MB in the production cases). The retry then abandons up to four of them per failing chunk, on the shared ssrf-safe dispatcher that every provider media read uses, and on the recovering path the upload continues afterwards rather than dying, so the sockets accumulate inside posts that go on to publish.
Went with response.body?.cancel() rather than the drain-by-reading idiom used in SocialAbstract.fetch (await request.text()), because buffering a 771 MB error body would reintroduce the memory behaviour the streaming refactor removed.
Measurable in an end-to-end run against a storage server injecting the fault: before the fix retries consistently waited the full backoff, after it several came back in 0.2-0.7s, so the pool was genuinely stalling. The post still recovers from the injected 200s and publishes.
A non-206 response is discarded without consuming its body, so undici keeps the socket checked out. The body here can be the whole object, and a retry abandons up to four of them per chunk on a shared dispatcher.
1 participant
What kind of change does this PR introduce?
Bug fix in the orchestrator/provider layer (media reads shared by all chunked uploaders).
youtubeChunkStream,tiktokChunkStreamandSocialAbstract.mediaChunkall read the stored video back in ranged GETs and threw a non-retryableBadBodythe moment the store answered anything other than206. A single such answer failed the whole post. They now retry the identical range up to three times (the existingtotalRetries/timer(5000)recursion used bySocialAbstract.fetchandrunStreamedUpload) before giving up, and record the real{status, statusText, ok}in the failure details instead of a hardcoded'{}'.youtubeMediaSizegets the same treatment for a HEAD with noContent-Length.What deliberately stayed the same: the refusal itself. A non-206 body is still never fed into the upload, and once the retries are spent the failure is still a terminal
BadBody, so genuinely broken media fails exactly as before, only with a diagnosable error.Why was this change needed?
The customer case
A Postiz Cloud customer had 15 of 15 scheduled YouTube posts fail between Aug 27 and Sep 1, every one with
The media storage did not return the requested byte range, please try againthrown fromyoutubeChunkStream->finalizePost. 14 of them published successfully after simply restarting the unchanged row — same media, thumbnail, description and credentials. Their own client-side checker read the same objects successfully, with correct206andContent-Range, immediately before each restart. Two more rows from Aug 14 show the identical signature.It is prod-wide, and #1853 did not fix it
Querying
Post.error LIKE '%youtubeChunkStream%'over the last 30 days: 5-10 failed posts per day, across 2-6 organizations, every single day. The rate is flat across the merge of #1853 (2026-08-07,d1765d40), which addedaccept-encoding: identityto these reads on the theory that a compressed response loses itsContent-Lengthand Cloudflare then answers a range request with the whole object. That is deployed and the failures continued at the same rate, so the header was not sufficient on its own.We also could not tell what was actually happening, because the
BadBodyjson argument was hardcoded'{}'— and that argument is the only field that carries response diagnostics intoPost.error/Errors.message. All 15 customer rows decode to{"identifier":"youtube","json":"{}","body":"{}"}. Nothing reaches Sentry either:post.activity.tsonly logs webhook failures, so a non-retryableBadBodygoes straight tochangeState(...ERROR...)and is never reported.Reproduced directly against the CDN
Issuing the worker's exact request shape (
Range: bytes=<8 MiB window>,accept-encoding: identity, nothing else) against the hosted object from one of the failed rows, two of eight consecutive requests returned HTTP 200 carrying the complete 771 MB object — noContent-Range,cf-cache-status: BYPASS— with correct206responses seconds either side from the same POP. So Cloudflare in front of R2 intermittently ignoresRangeand answers with the whole file. Not offset-specific, not file-specific, not reproducible on demand.The fault lives only on objects too large for the cache
A sweep of 2,900 distinct production mp4 objects (ranged GET, identity encoding) found exactly three with
cf-cache-status: BYPASS— at 641.2 MB, 622.6 MB and 618.4 MB. Every other object returnedMISS(1636),REVALIDATED(650) orHIT(211), the largest cacheable one being 299.6 MB. So the cacheability cliff sits between 299.6 MB and 618.4 MB, which brackets Cloudflare's documented maximum cacheable file size of 512 MB on Free/Pro/Business (5 GB on Enterprise).Sampling with 8 MiB windows, every 200 we have ever observed was on a
BYPASSobject:Aggregate: 1 fault in 876 requests on BYPASS objects (~0.11%), 0 in ~3,000 on cacheable ones. The rate is bursty rather than steady — the reproduction above hit 2 in 8 (25%) on a different day, two orders of magnitude higher. At the 0.11% rate a 771 MB video (92 chunks) still has roughly a 10% chance of losing at least one chunk; during a burst it is near-certain. That is consistent with 5-10 failed posts a day.
Retrying is safe here because the YouTube path is a resumable upload session:
finalizePostprobesprobeUploadSessionfor the committed offset and resumes from exactly there, so re-reading a range cannot duplicate or corrupt a video. TikTok has no such probe, which is why its retry stays inside the read and still ends inBadBody— the caller rethrows rather than waiting on TikTok for a verdict on bytes that never arrived.QA
Verified end to end against a real connected YouTube channel, driving real posts through backend -> Temporal -> YouTube, with a local storage server standing in for the CDN so the 200-with-full-object answer can be injected on demand. To reproduce:
200with the full object and noContent-Range(the exact shape Cloudflare returns).PUBLISHED. In the storage log each injected200is followed ~5 s later by a repeat request for the identical byte range that returns206, and the upload continues from there. Observed: 4 injected 200s, all recovered, post published.200and the full object, and schedule another post. Expected: exactly 4 attempts on the same range (initial + 3 retries) roughly 5 s apart, then the post goes toERROR. Observed as described.select error from "Post" where id = '<the post id>';. Expected: the details now contain{"status":200,"statusText":"OK","ok":true}instead of the previous empty{"json":"{}"}, naming the answer the store actually gave.8c4d5113) and confirm recovery still works. Expected: the post publishes as before, and the retry following each injected200now returns quickly rather than always waiting the full backoff, since the socket is released instead of being held by the abandoned response. Observed: 5 injected 200s, all recovered, several retries answered in 0.2-0.7 s.Other information:
uploads.postiz.comalready sendscache-control: max-age=14400on these responses and also carries a straylocation: https://postiz.com/api/successheader on both 200 and 206 answers, which suggests a rule sitting in front of the bucket.{status, statusText, ok}diagnostics on their own. That work is folded in here as an exportedrangeReadFailurehelper shared by all three read sites, so fix(media): surface the real status when a ranged media read is not 206 #1932 can be closed.accept-encoding: identityacross the shared helpers a day later and missed them, fix(youtube,tiktok): request identity encoding on the media reads #1853 added it to them. A longer-term cleanup would be deletingyoutubeMediaSizein favour ofSocialAbstract.mediaSizeand consolidating both chunk readers into a shared ranged-stream helper, which is the one shapeSocialAbstractdoes not currently offer.