Run gRPC deterministically in simulation and test the worker ControlService by saintstack · Pull Request #14337 · apple/foundationdb

saintstack · 2026-10-21T06:12:04Z

This PR shows that gRPC can run under FDB's deterministic simulation and be tested under faults. It makes FDB build the gRPC it tests.

Run real gRPC deterministically inside FDB's simulator, build the gRPC that FDB tests, and serve and test the worker ControlService in simulation. Design: design/grpc-simulation.md.

Until now, gRPC services in fdbserver had no simulation coverage. gRPC brings its own threads, reads the real clock, seeds its random generators from the OS and orders some work by pointer value, so simulated processes never started it. FDB also linked whatever gRPC was installed in /opt/grpc.

Changes

SimEventEngine (fdbrpc/SimEventEngine.cpp): a gRPC EventEngine that runs gRPC's I/O, timers and closures on the Flow run loop over INetworkConnections. Under Sim2, real gRPC then runs on simulated sockets and simulated time.
Vendored gRPC (cmake/CompileGrpc.cmake): gRPC 1.83.0 with its bundled abseil, protobuf, re2 and c-ares, built as targets in FDB's build graph. It replaces the /opt/grpc search. GRPC_SOURCE_DIR supports hosts without GitHub access. Three patches (cmake/patches/grpc/) make gRPC deterministic:
- insertion-ordered queued-call sets;
- chttp2 stream cancellation in stream-id order;
- an optional abseil seed-material source, set only by simulation.
ControlService in simulation:
- ControlService becomes a callback service whose handlers run Flow work on the hosting process.
- With the test config grpcEnabled, every simulated fdbd process serves it.
- New workloads and tests: GrpcControlService, GrpcEcho, and the /fdbrpc/simgrpc unit tests.
Fixes:
- GrpcServer's stop path was a no-op (inverted null check).
- GrpcServer::run logged a SevError on actor_cancelled, and the exclude handlers swallowed it.
- Worker gRPC addresses were never persisted. The worker list is now encoded at the 8.0 protocol version, which 7.4 and 8.0 can both read.
- Sim2 reported a spurious leaked connection when a peer closed after its owner.
- An unused variable in FlowGrpcTests.cpp broke -Werror builds now that gRPC code always compiles.

Testing

Determinism: each seed was run twice, with different working directories and environment sizes, on macOS and Linux, and gave identical results.
Joshua, on the pre-rebase version of this change:
- all tests, 100k runs: 100,000 passed;
- an earlier all-tests run of 100k: no failures, 3 runs killed externally;
- GrpcControlService alone, 100k runs: 100,000 passed;
- the restarting/to_* downgrade tests passed.
Since then: rebased onto current main, plus two -Werror fixes. Every commit of the series builds on Linux. A -Werror Linux build passes.

20261021-060748-stack-echo-6246e2c69d991d47 compressed=True data_size=42295071 duration=3895412 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:34:44 sanity=False started=100000 stopped=20261021-064232 submitted=20261021-060748 timeout=5400 username=stack-echo

Cost

Clean Linux build (ninja fdbserver, -j6): about 7 minutes (+29%) over building against the prebuilt /opt/grpc. Stripped fdbserver is +12.7 MB compared with WITH_GRPC=OFF. Incremental builds are unchanged.

For reviewers

This makes FDB own gRPC 1.83 plus three patches, which have to be re-applied on gRPC upgrades.
Known limitations are listed in the design doc:
- gRPC hangs under small memory quotas;
- simulated reboots leak gRPC servers;
- the blocking *Sync stop paths.

@ploxiln Interested in your view of this vendoring of grpc et al.

…ervice gRPC services in fdbserver had no simulation coverage: gRPC brings its own threads, reads the real clock, seeds from the OS and orders some work by pointer value, so simulated processes never started it. FDB also linked whatever gRPC was installed in /opt/grpc. - SimEventEngine runs gRPC's I/O, timers and closures on the Flow run loop over INetworkConnections, so under Sim2 real gRPC runs on simulated sockets and time. - gRPC 1.83.0 is built from source in FDB's build graph (cmake/CompileGrpc.cmake) with three patches: insertion-ordered queued-call sets, chttp2 stream cancellation in stream-id order, and an optional abseil seed-material source set only by simulation. - ControlService becomes a callback service whose handlers run Flow work on the hosting process. With the test configuration grpcEnabled, every simulated fdbd process serves it; the GrpcControlService and GrpcEcho workloads and the /fdbrpc/simgrpc unit tests exercise gRPC under faults. - Fixes: the gRPC server's stop path was a no-op; GrpcServer::run logged a SevError on actor_cancelled and the exclude handlers swallowed it; worker gRPC addresses were never persisted (the worker list is now encoded at 8.0, readable by 7.4 and 8.0); Sim2 reported a spurious leaked connection when a peer closed after its owner. See design/grpc-simulation.md.

gRPC 1.83 deprecates the TLS credential APIs FDB used; GCC's -Wno-deprecated does not cover deprecated declarations, so -Werror failed. Port Credentials.h to InMemoryCertificateProvider, set_root/identity_certificate_provider and TlsServerCredentialsOptions::Create (equivalent: the legacy provider plus watch flags set the same root and identity providers), and include gRPC's, abseil's and protobuf's headers as system headers, as a prebuilt package's are. Skip /fdbrpc/simgrpc in fdbrpc_test's normal (non-simulated) mode.

ploxiln · 2026-10-21T16:32:34Z

+  if(GRPC_SOURCE_DIR)
+    grpc_run(copy ${CMAKE_COMMAND} -E copy_directory "${GRPC_SOURCE_DIR}" "${GRPC_SRC}")
+  else()
+    grpc_run(clone ${GIT_EXECUTABLE} clone --quiet --depth 1 --branch v${GRPC_VERSION}
+             https://github.com/grpc/grpc.git "${GRPC_SRC}")
+    grpc_run(submodules ${GIT_EXECUTABLE} -C "${GRPC_SRC}" submodule update --init --depth 1
+             third_party/abseil-cpp third_party/protobuf third_party/re2 third_party/cares/cares)


If you can, I suggest replacing all the git cloning and patching with ExternalProject_add(), for an example see:

foundationdb/cmake/awssdk.cmake

Lines 20 to 34 in 1915da1

ExternalProject_Add(awssdk_project

GIT_REPOSITORY https://github.com/aws/aws-sdk-cpp.git

GIT_TAG c4b8cb01b0215f00740d9d72f7185ee056ced1f6 # v1.11.473

GIT_SHALLOW ON

SOURCE_DIR "${CMAKE_CURRENT_BINARY_DIR}/awssdk-src"

BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/awssdk-build"

# it seems advice.detachedHead breaks something which causes aws sdk to always be rebuilt.

# This option forces to cmake to build the aws sdk only once and never attempt to update it

UPDATE_DISCONNECTED ON

PATCH_COMMAND # enable s2n for macOS too

sed -i.bak -e "s/UNIX AND NOT APPLE AND NOT BYO_CRYPTO/UNIX AND NOT BYO_CRYPTO/"

crt/aws-crt-cpp/CMakeLists.txt

COMMAND # patch cmake_minimum_required in s2n sub-sub-module

sed -i.bak -E -e "s/\$VERSION 3.0\$/\$VERSION 3.10\$/"

crt/aws-crt-cpp/crt/s2n/CMakeLists.txt

The downside is:

it can be quite difficult to define an appropriate PATCH_COMMAND that performs robustly, especially for download methods such as git where changing the GIT_TAG will not discard changes from a previous patch, but the patch command will be called again after updating to the new tag

So the simple thing is to just manually blow the whole grpc source checkout away, and let it clone and patch from scratch. But for the relative simplicity of using the more stock CMake functionality, I think it's worth it. Most devs will not be modifying the grpc dependency here.

In the future I have an idea to use a custom DOWNLOAD_COMMAND that will download and cache tarballs, and extract in the appropriate order and subdirs, instead of recursive git clone.

FetchContent_makeAvailable is also good, maybe better

Did Fetch_Content after chatting w/ @ploxiln

xis19 · 2026-10-21T16:58:30Z

Personally I am pretty against downloading from external websites during the build process. In this case we are much more vulnerable to mitm or upstream library poisoning.

saintstack · 2026-10-21T17:41:14Z

Personally I am pretty against downloading from external websites during the build process. In this case we are much more vulnerable to mitm or upstream library poisoning.

Fair. But we currently download boost, rocksdb, etc. This download does deviate some downloading by tag... Should probably do as others do and use a hash... Thanks @xis19

ploxiln · 2026-10-21T18:01:09Z

+  # A directory without a CMakeLists.txt, so FetchContent_MakeAvailable only populates; add_grpc() adds the project
+  # with EXCLUDE_FROM_ALL, which FetchContent supports only from CMake 3.28.
+  SOURCE_SUBDIR fdb-populate-only)


Our rockylinux9 build has cmake 3.31, so I think it would be fine to require cmake 3.28 for main (and even release-8.0 branch)

ploxiln · 2026-10-21T18:08:20Z

+  PATCH_COMMAND ${GIT_EXECUTABLE} reset --quiet --hard --recurse-submodules
+  COMMAND ${GIT_EXECUTABLE} clean --quiet -fd
+  COMMAND ${GIT_EXECUTABLE} submodule --quiet foreach ${GIT_EXECUTABLE} clean --quiet -fd
+  COMMAND ${GIT_EXECUTABLE} apply --whitespace=nowarn ${GRPC_PATCHES}


Performing patch step for 'grpc-populate' C:/Program Files/Git/bin/git.exe: line 1: C:/Program: No such file or directory fatal: run_command returned non-zero status for third_party/abseil-cpp

I think we could get the windows cmake setup to work if we quoted the GIT_EXECUTABLE like "${GIT_EXECUTABLE}"

(or maybe just disable GRPC in the windows cmake setup CI job)

... another plausible option is to just assume "git" binary is available:

Suggested change

COMMAND ${GIT_EXECUTABLE} apply --whitespace=nowarn ${GRPC_PATCHES}

COMMAND git apply --whitespace=nowarn ${GRPC_PATCHES}

foundationdb-ci · 2026-10-21T18:17:43Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 7ffecce
Duration 0:43:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-10-21T18:27:42Z

Result of foundationdb-pr-macos-m1 on macOS 14.x

Commit ID: 7ffecce
Duration 0:53:46
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-10-21T18:32:34Z

Result of foundationdb-pr-clang-arm on Linux RHEL 9

Commit ID: 7ffecce
Duration 0:58:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-10-21T18:40:07Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 7ffecce
Duration 1:06:09
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-10-21T18:45:38Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 7ffecce
Duration 1:11:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-10-21T18:48:02Z

Result of foundationdb-pr-macos on macOS 14.x

Commit ID: 7ffecce
Duration 1:14:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

ploxiln · 2026-10-21T18:50:57Z

Should probably do as others do and use a hash

Agreed, that helps a lot. sha1 is considered weak, but git hashes are still "secure" today. In the future some cached tarball downloads protected by sha256 will be nice.

foundationdb-ci · 2026-10-21T19:31:39Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 7ffecce
Duration 1:57:44
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack requested a review from ploxiln October 21, 2026 06:13

saintstack marked this pull request as ready for review October 21, 2026 06:16

saintstack requested a review from brownleej October 21, 2026 06:19