[johscheuer]

In some cases it can happen that the connection string contains an old coordinator address if using DNS entries in the cluster file those entries are resolved. If the old coordinator address is not resolvable anymore, e.g. because the Pod was deleted, this can cause the whole database creation to be stuck, even thought it fetches the updated connection string (and the file on disk was updated). This makes the whole database context unusable and the only way out of this is to update the seed connection string and restart the process. We observed that behaviour in one of our operator e2e tests. I created a new simulation test for this which failed without the fix:

$ ./packages/bin/fdbserver -r simulation -f ../src/foundationdb/tests/fast/StaleCoordinatorProtocolMonitor.toml -b off
...
Run test:StaleCoordinatorProtocolMonitor Done.

  0 tests passed; 1 tests failed.

  Unseed: 67039
  Elapsed: 1095.412801 simsec, 49.243298 real seconds
  1 SevError events logged

The failure was:

<Event Severity="40" ErrorKind="Unset" Time="1095.199101" DateTime="2026-10-20T10:38:05Z" Type="StaleCoordinatorProtocolMonitorResult" Machine="3.4.3.4:1" ID="0000000000000000" NumDatabases="20" NumStaleCoordinators="9" CompletedDatabases="2" StuckDatabases="18" ThreadID="8941379931926674692" Backtrace="/usr/bin/addr2line -e ./packages/bin/fdbserver -p -C -f -i 0x45f8d56 0x45f70c1 0x45f726d 0x16e7774 0x271a613 0x271ab47 0x26ff63d 0x26f79a0 0x270cffc 0x1051941 0x3b74399 0x3b74700 0x3c89cb0 0x9a26a8 0x7f72c5332610" LogGroup="default" Roles="TS" />

and with the fix the test passed as expected (I tested with -b off and -b on:

$ ./packages/bin/fdbserver -r simulation -f ../src/foundationdb/tests/fast/StaleCoordinatorProtocolMonitor.toml -b off
...
Run test:StaleCoordinatorProtocolMonitor Done.

  1 tests passed; 0 tests failed.

  Unseed: 72090
  Elapsed: 43.530260 simsec, 1.692274 real seconds

I'm currently running a simulation run with Joshua and will report back once the result is available. Result:

All ensembles:
  20261020-105841-jscheuermann-bca9b0eb57c1823d      compressed=True data_size=43211693 duration=3994099 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:36:26 sanity=False started=100000 stopped=20261020-113507 submitted=20261020-105841 timeout=5400 username=jscheuermann

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: 7f74153
• Duration 0:22:09
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 7f74153
• Duration 0:33:59
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 7f74153
• Duration 0:46:47
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux RHEL 9

• Commit ID: 7f74153
• Duration 0:47:48
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 7f74153
• Duration 1:42:09
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 7f74153
• Duration 0:41:38
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 7f74153
• Duration 0:52:50
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)