[9.3.0] remote: add gRPC download idle timeout - #30667
Open
sluongng wants to merge 1 commit into
Open
Conversation
sluongng
force-pushed
the
sluongng/cherrypick-29916-9.3.0
branch
from
August 11, 2026 12:46
96bd345 to
32ea41e
Compare
iancha1992
changed the title
iancha1992
added
team-Remote-Exec
Member
|
@sluongng Is this ready for review? |
Contributor
Author
|
it is, but it's based on #30666 so i was hoping to get that one merged first. i will just flip the switch anyway then |
sluongng
marked this pull request as ready for review
August 14, 2026 09:11
github-actions
Bot
added
the
team-Core
iancha1992
enabled auto-merge
August 14, 2026 18:18
Member
|
@sluongng I think this one is now ready to merge. Could you please resolve the conflicts? |
auto-merge was automatically disabled
August 18, 2026 23:07
Head branch was pushed to by a user without write access
sluongng
force-pushed
the
sluongng/cherrypick-29916-9.3.0
branch
from
August 18, 2026 23:07
b2aafe3 to
9ff3fd5
Compare
The gRPC service-config support from bazelbuild#29912 is now on master. It lets users lengthen the whole-call deadline for `ByteStream.Read` so large downloads can keep running while they make progress. A longer deadline also lets a stalled stream wait just as long, because a total RPC deadline cannot distinguish progress from inactivity. This adds `--remote_grpc_download_idle_timeout` as an independent inactivity limit for `ByteStream.Read`. It defaults to 60 seconds to match the existing HTTP remote-cache default; setting it to `0` disables it. When a read goes idle, Bazel cancels it and translates the timeout-owned `CANCELLED` closure to `DEADLINE_EXCEEDED` so the normal cache retrier can resume from the received offset. Other RPCs, including `ByteStream.Write`, `Execute`, and `WaitExecution`, are unaffected. Timeout tracking uses a monotonic deadline and at most one scheduled task per call. This avoids retaining a canceled task for every response and prevents a timer from canceling a read that has since made progress. Normal gRPC forwarding stays in `finally` blocks so timeout bookkeeping failures cannot suppress messages or callbacks. Tests cover: - the 60-second option default and exclusion of `ByteStream.Write`; - cancellation status and single-task scheduling as reads make progress; - request and response forwarding when timeout bookkeeping fails; and - an in-process `ByteStream` flow that sends a prefix, stalls, fires the timeout deterministically, and verifies that the retry resumes from the next offset. The interceptor currently applies only to `ByteStream.Read`. The download-specific option name leaves room to cover future server-streaming download RPCs such as [`SplitChunks`](bazelbuild/remote-apis#377) without implying that every remote stream should share this timeout. Closes bazelbuild#29916. PiperOrigin-RevId: 956359263 Change-Id: Id8ae7c6dbb172f1e7e799536193458d9cbfb8c0c (cherry picked from commit 6e27dd1)
sluongng
force-pushed
the
sluongng/cherrypick-29916-9.3.0
branch
from
August 18, 2026 23:45
9ff3fd5 to
73f950e
Compare
iancha1992
enabled auto-merge
August 19, 2026 17:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gRPC service-config support from #29912 is now on master. It lets users lengthen the whole-call deadline for
ByteStream.Readso large downloads can keep running while they make progress. A longer deadline also lets a stalled stream wait just as long, because a total RPC deadline cannot distinguish progress from inactivity.This adds
--remote_grpc_download_idle_timeoutas an independent inactivity limit forByteStream.Read. It defaults to 60 seconds to match the existing HTTP remote-cache default; setting it to0disables it. When a read goes idle, Bazel cancels it and translates the timeout-ownedCANCELLEDclosure toDEADLINE_EXCEEDEDso the normal cache retrier can resume from the received offset. Other RPCs, includingByteStream.Write,Execute, andWaitExecution, are unaffected.Timeout tracking uses a monotonic deadline and at most one scheduled task per call. This avoids retaining a canceled task for every response and prevents a timer from canceling a read that has since made progress. Normal gRPC forwarding stays in
finallyblocks so timeout bookkeeping failures cannot suppress messages or callbacks.Tests cover:
ByteStream.Write;ByteStreamflow that sends a prefix, stalls, fires the timeout deterministically, and verifies that the retry resumes from the next offset.The interceptor currently applies only to
ByteStream.Read. The download-specific option name leaves room to cover future server-streaming download RPCs such asSplitChunkswithout implying that every remote stream should share this timeout.Cherry-pick of #29916. Stacked on #30666, which backports #29912.